An augmented reality multi-modal information processing method and system for a diagnosis and treatment scene

Abstract

The application provides a kind of diagnosis and treatment scene-oriented augmented reality multi-modal information processing method and system, it is related to augmented reality information processing technical field, including: in response to interactive trigger operation, generate the diagnosis and treatment of target user associated identification information;Determine the dynamic pose information of target user in monitoring space based on the diagnosis and treatment of identification information and multi-source perception data;Dynamic pose information is aligned with time sequence record data space-time dimension, constructs multi-modal feature set;Multi-modal feature set is input into the feature inference of inference model based on priori knowledge graph constraint, generates intermediate characteristic analysis result;In response to intermediate characteristic analysis result, dynamically adjust the labeling display area of dynamic pose information, generate augmented instruction signal;According to augmented instruction signal and pre-set real-time feedback data, the intermediate characteristic analysis result is closed-loop corrected, and output visual rendering display data.The application solves the accuracy and reliability problem of diagnosis and treatment site analysis.

Description

Technical Field

[0001] This invention relates to the field of augmented reality information processing technology, and more specifically, to an augmented reality multimodal information processing method and system for medical scenarios. Background Technology

[0002] In existing augmented reality (AR) information processing technologies, terminals typically collect first-person perspective images, videos, and audio, and display target identity information and auxiliary analysis results to the user. However, limited by processor performance, power consumption, and heat dissipation capabilities, AR terminals struggle to handle complex image analysis, multimodal model inference, and other computationally intensive tasks for extended periods, thus these tasks are usually delegated to servers. However, existing server-side analysis systems often treat images, audio, or text as independent data, lacking real-time correlation analysis between on-site multimodal data and the target user's historical data. Furthermore, their returned results are often plain text or interface information, making it difficult to effectively integrate with the spatial positioning, real-time display, and interactive operations of the AR terminal. This frequently results in inaccurate analysis results, missing information, or disconnect from the current scene, impacting the accuracy and reliability of on-site medical analysis.

[0003] Therefore, there is an urgent need for an augmented reality multimodal information processing method and system for diagnosis and treatment scenarios, which solves the problems of accuracy and reliability in on-site analysis of diagnosis and treatment. Summary of the Invention

[0004] The purpose of this invention is to provide an augmented reality multimodal information processing method and system for clinical scenarios, in order to improve the aforementioned problems. To achieve the above objective, the technical solution adopted by this invention is as follows:

[0005] Firstly, this application provides an augmented reality multimodal information processing method for diagnostic and treatment scenarios, including:

[0006] In response to interactive triggering operations, generate consultation identification information associated with the target user;

[0007] Based on the consultation identification information and the multi-source perception data of the collected users, the dynamic pose information of the target user in the monitoring space is determined.

[0008] The dynamic pose information is aligned with the pre-stored time-series recording data in the spatiotemporal dimension to construct a multimodal feature set;

[0009] The multimodal feature set is input into a reasoning model constrained by prior knowledge graphs for feature reasoning, generating intermediate feature analysis results for interface annotation.

[0010] In response to the intermediate feature analysis results, the annotation display area of ​​the dynamic pose information is dynamically adjusted to generate an enhanced indication signal;

[0011] The intermediate feature analysis results are corrected in a closed loop based on the enhanced indication signal and preset real-time feedback data, and the data is then output as a visual rendering display.

[0012] Secondly, this application also provides an augmented reality multimodal information processing system for medical scenarios, including:

[0013] The interaction module is used to generate consultation identification information associated with the target user in response to interactive triggering operations;

[0014] The pose module is used to determine the dynamic pose information of the target user in the monitoring space based on the consultation identification information and the multi-source perception data of the collected users.

[0015] The alignment module is used to align the dynamic pose information with the pre-stored time-series recording data in the spatiotemporal dimension to construct a multimodal feature set;

[0016] The reasoning module is used to input the multimodal feature set into a reasoning model based on prior knowledge graph constraints for feature reasoning and generate intermediate feature analysis results for interface annotation.

[0017] An adjustment module is used to dynamically adjust the annotation display area of ​​the dynamic pose information in response to the intermediate feature analysis results, and generate an enhanced indication signal.

[0018] The correction module is used to perform closed-loop correction on the intermediate feature analysis results based on the enhanced indication signal and preset real-time feedback data, and output visualized rendering display data.

[0019] The beneficial effects of this invention are as follows:

[0020] This invention aligns real-time dynamic pose with time-series recorded data in a spatiotemporal dimension, unifying the time reference and spatial coordinate system, thus eliminating spatiotemporal biases in heterogeneous sensor data. Based on this, a reasoning model constrained by prior knowledge graphs removes abnormal reasoning results that violate clinical rules, ensuring that the intermediate feature analysis results align with diagnostic logic and reducing the probability of errors in augmented reality terminal visualization information. Subsequently, the annotation display area is adjusted according to the target user's dynamic pose based on the intermediate feature analysis results, and the augmented indication signal adapts to the human posture in real time, resolving annotation occlusion and misalignment issues and ensuring clear and intuitive access to augmented reality terminal prompts. Finally, closed-loop correction continuously offsets errors caused by sensor noise and pose calculation, guaranteeing the accuracy of the visualization information output. In summary, this invention solves the problems of accuracy and reliability in on-site diagnostic analysis.

[0021] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing embodiments of the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the augmented reality multimodal information processing method for diagnosis and treatment scenarios described in this embodiment of the invention;

[0024] Figure 2 This is a schematic diagram of the structure of the augmented reality multimodal information processing device for diagnosis and treatment scenarios described in this embodiment of the invention.

[0025] The diagram is labeled as follows: 800, Augmented Reality Multimodal Information Processing Device for Diagnosis and Treatment Scenarios; 801, Processor; 802, Memory; 803, Multimedia Component; 804, I / O Interface; 805, Communication Component. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0027] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0028] In current clinical settings, doctors wear augmented reality (AR) devices to collect patients' visual and audio data. However, due to limited local computing power, the multimodal data needs to be uploaded to a server for processing. Existing server systems lack deep integration between the perceived data and the patient's historical electronic medical records. The returned text results cannot be effectively integrated with the AR display coordinate system, leading to display drift, occlusion, or disconnection from the field of vision when the patient or doctor moves their head. Furthermore, the analysis results generated by large models often lack dynamic prompts optimized for individual medical history and current vital signs, as well as result verification modules, resulting in inaccuracies or inconsistencies with individual circumstances. These multiple factors collectively reduce the accuracy and reliability of doctors' judgments based on AR information in clinical settings.

[0029] Example 1:

[0030] This embodiment provides an augmented reality multimodal information processing method for diagnostic and treatment scenarios.

[0031] See Figure 1 The figure shows that the method includes steps S1 to S6, including:

[0032] S1: In response to interactive triggering operations, generate consultation identification information associated with the target user;

[0033] In this step, in response to the interactive trigger operation, a communication connection is established between the augmented reality (AR) terminal and the server, and the doctor identifier, patient identifier, terminal identifier, and terminal time information are sent to the server. The server generates a session identifier for this consultation and establishes the association between the doctor, patient, augmented reality terminal, and consultation session, obtaining the consultation identifier information.

[0034] Among them, the session identifier is a unique identifier generated by the server for a continuous diagnosis and treatment process. It is used to distinguish different diagnosis and treatment processes for the same patient and to associate the images, audio, text, model results and doctor correction data generated in this diagnosis and treatment.

[0035] The doctor identifier is a unique identifier for the current doctor in the medical information, used to record the data access subject, the subject operating the large model results, and the source of the correction data;

[0036] The patient identifier is a unique identity identifier corresponding to the current target user in the medical information system. It can be obtained through registration information, electronic medical records, QR codes, near-field communication, or doctor confirmation. It is used to retrieve and associate the target user's basic information and historical medical data. The server retrieves the target user's basic information and historical medical data from the patient database based on the patient identifier. The historical medical data includes at least one of the following: past medical history, historical examination information, historical images, historical medical records, and medication records.

[0037] The terminal identifier is a unique device identifier corresponding to the current augmented reality terminal, used to determine the source of multimodal data collection at the diagnosis and treatment site;

[0038] Terminal time information is a data time reference jointly used by the augmented reality terminal and the server, and is used to correlate the time of collected images, audio, text and interactive events.

[0039] The expression for the consultation identification information is:

[0040] ;

[0041] In the above formula, For consultation identification information, For session identification, For doctors' identification, Identify patients, To enhance the identification of real-world terminals, To enhance the unified time reference used by both the terminal and the server.

[0042] S2: Determine the dynamic pose information of the target user in the monitoring space based on the consultation identification information and the multi-source perception data of the collected users;

[0043] To clarify the specific method for acquiring dynamic pose information, step S2 includes S21 to S24, specifically:

[0044] S21: Based on the consultation identification information, retrieve the time-series image, collect the visual data of the target user in the time-series image, and obtain multi-source perception data;

[0045] In this step, the consultation identification information is used as a data encapsulation tag to retrieve the time-series images or video frames of the consultation site, and simultaneously combine the consultation voice, response voice and operation voice commands of the target user in the time-series images into an audio data stream.

[0046] When multiple human targets exist within the observation area of ​​the time-series image, the human body with the largest area of ​​the image is automatically selected as the candidate target user. Visual data is then collected from the candidate target user to obtain multi-source perception data. The candidate target user can also be confirmed and locked with one click via voice or gesture to avoid confusion of the tracking object.

[0047] The visual data includes the target user's face, tongue, lesions on the body surface, and other observation areas.

[0048] S22: Extract local feature points from the multi-source sensing data, and filter valid matching points by matching the feature points of the current frame with the feature points of the previous frame;

[0049] In this step, local features are extracted from the multi-source sensing data, the target region feature point set cached in the previous frame is read, and coarse matching of features between the previous and next frames is performed using Euclidean distance metric. Then, abnormal matching pairs that have offset, scale distortion, or mismatch are eliminated by geometric consistency constraints. The remaining feature pairs that meet the distance threshold and geometric constraint conditions are counted as valid matching points. At the same time, the number of valid matching points is recorded for subsequent tracking stability determination.

[0050] S23: Calculate the pose transformation within the time-series image based on the effective matching points, and estimate the pose state by combining the preset inertial sensing data to obtain the actual position change of the target user.

[0051] In this step, the translation, scaling, and rotation pixel pose transformations of the target region between adjacent image frames in the time-series image are calculated based on the effective matching points. At the same time, the inertial sensing data output by the gyroscope and accelerometer built into the augmented reality (AR) terminal are read to calculate the global motion components of the screen caused by head rotation and terminal displacement. The global motion components are removed from the feature-matched region transformation by differential compensation to eliminate the screen offset interference caused by the user's own actions. Finally, the relative displacement value generated only by the target user's body movement is separated, which is the true position change of the target user relative to the AR field of view.

[0052] S24: Update the pose estimation of the current frame based on the actual position change. When the number of valid matching points is lower than the preset matching threshold, re-execute the person detection to generate real-time spatial positioning and posture data of the user as dynamic pose information.

[0053] In this step, the image pose estimation coordinates of the target user in the current frame are iteratively updated based on the actual position change. When the number of effective matching points is higher than a preset matching threshold, the image coordinate system is mapped to the augmented reality (AR) display coordinate system using pre-calibrated camera intrinsic and extrinsic parameters and inertial sensor attitude data. The target user's positioning data is then output as the dynamic pose information for this frame. When the number of effective matching points or the real-time positioning confidence is detected to be lower than the matching threshold, the feature tracking process is interrupted, the target user in the image is re-identified, and the observation area is locked to correct tracking failures caused by tracking loss, occlusion, or large patient movements. The generated real-time spatial positioning and attitude data of the user are continuously updated frame by frame and ultimately used as the dynamic pose information.

[0054] The location data of the target user includes the target unique ID, real-time ROI region, effective feature set, location confidence and frame timestamp location data.

[0055] S3: Align the dynamic pose information with the pre-stored time-series recording data in the spatiotemporal dimension to construct a multimodal feature set;

[0056] To clarify the specific method for obtaining the multimodal feature set, step S3 includes S31 to S34, specifically:

[0057] S31: Perform inter-frame feature matching and attitude compensation calculation on the dynamic pose information to obtain target user positioning data;

[0058] In this step, the dynamic pose information is read, and further, local features of the target user in the preceding and following frames are retrieved from the dynamic pose information for inter-frame feature matching. Global image pose motion compensation is performed using the inertial attitude data output by the AR terminal's inertial sensor to eliminate interference from head rotation and terminal displacement causing overall field of view shift. Valid matching feature points are selected through geometric constraints to calculate the target user's regional translation, scale, and rotation changes, ultimately generating the target user's positioning data.

[0059] The expression for the target user location data is:

[0060] ;

[0061] In the above formula, Targeting data for specific users The target identifier for the currently tracked target user. The target user's location within the current frame. For continuous tracking of local features of target users, Determine the location reliability for the target user. This is the data timestamp.

[0062] S32: Synchronize and align the target user location data with the time-series recording data to form a time-aligned sequence;

[0063] In this step, the time reference of the consultation identification information is used as the alignment benchmark. The timestamp carried by the target user's location data is extracted, and the time-series record data cached and pre-stored on the server is retrieved. The location data, image frames and audio segments that are successfully matched within the same time window are matched to ensure that the visual, audio and location data are synchronized in the time dimension. This avoids time misalignment between the consultation voice and the target user's visual image and target user's location information, and finally forms a time-aligned sequence.

[0064] The time-series recorded data includes images or videos and audio uploaded in real time by the augmented reality (AR) terminal, with added session identifiers, patient identifiers, data type identifiers, and timestamps.

[0065] The expression for the time-aligned sequence is:

[0066] ;

[0067] In the above formula, For time-aligned sequences, Session identifier; The target identifier of the currently tracked target user; Augmented reality terminal identification; Image data or video data; Audio data; Data timestamp.

[0068] S33: Extract the relative position change in the target user's location data from the time alignment sequence, and perform spatial coordinate mapping by combining it with a preset historical spatial registration reference image to obtain spatial registration features;

[0069] In this step, the change in the center point coordinates of the target user's location data is extracted from the time alignment sequence. The target user region in the historical spatial registration reference image plane is transformed into the target user display region corresponding to the augmented reality (AR) real display space. Then, spatial coordinate mapping and viewpoint unification calibration are performed based on the target user display region to finally generate spatial registration features.

[0070] S34: The spatial registration features and the preset historical record features are fused to form a multimodal feature set.

[0071] In this step, the historical record features include preprocessed visual data, speech-to-text medical text, basic information of the target user, and previous medical history data. The spatial registration features are used as spatial location dimension features and are fused with the historical record features through a unified data field specification. Session identifiers, data source indexes, and unified timestamps are bound simultaneously to construct a complete multimodal context-based multimodal feature set. The multimodal feature set carries the target user's real-time spatial location registration information, on-site medical visual and audio information, and the target user's full-cycle medical history information, realizing the unified encapsulation of features from spatial positioning information, real-time medical perception data, and the target user's historical medical data.

[0072] The expression for the multimodal feature set is:

[0073] ;

[0074] In the above formula, For multimodal feature sets, For session identification, Identify patients, For preprocessed visual data, This is the medical text after audio processing. For the target user's basic information, For the target users' historical medical data, For data source indexing, This is the data timestamp.

[0075] S4: Input the multimodal feature set into the reasoning model based on prior knowledge graph constraints for feature reasoning, and generate intermediate feature analysis results for interface annotation;

[0076] The reasoning model is based on a hybrid reasoning architecture of domain knowledge graph and large language model (LLM), wherein the domain knowledge graph handles structured knowledge retrieval and logical constraint verification, and the large language model (LLM) handles unstructured text semantic understanding and flexible reasoning.

[0077] To clarify the specific method for obtaining the reasoning model based on prior knowledge graph constraints, step S4 includes S40 to S43, specifically:

[0078] S40: Based on a preset domain knowledge graph, perform entity alignment and attribute mapping on the multimodal feature set to obtain structured knowledge graph nodes;

[0079] In this step, based on the knowledge graph of traditional Chinese and Western medicine / clinical diagnosis and treatment scenarios, the entities of visual lesions, consultation symptoms, medical history of target users and test indicators in the multimodal feature set are aligned, and the duration of symptoms and other attributes are mapped to the standard fields of the graph. Finally, a structured knowledge graph node carrying association weights, timestamps and data source indexes is output.

[0080] S41: Apply loss penalty to inference results that do not meet the preset constraints based on the prior constraint regularization term to obtain intermediate results of canonical inference;

[0081] In this step, based on pre-set prior constraint regularization terms in the diagnosis and treatment domain, the constraint regularization terms include symptom-disease logic constraints, medication contraindication constraints, test indicator threshold constraints, and medical history time sequence logic constraints.

[0082] Based on prior constraint regularization terms, compliance comparisons are performed on several candidate inference branches in the large model. Each candidate branch includes a set of preliminary disease inferences, associated symptoms, suggested examination items, and medication recommendations. The server compares the structured content of each candidate inference branch with the prior constraint regularization terms. When a candidate inference branch exhibits disease-symptom associations that violate common sense in diagnosis and treatment, test indicators that exceed the physiologically reasonable range, suggested medications that conflict with the patient's medical history, or inference conclusions that significantly contradict the patient's historical treatment timeline, a penalty is imposed by amplifying the loss value of the regularization terms of that branch, thereby reducing the compliance score. Then, the branches are filtered and sorted according to the level of the penalty values. The branch with the lowest penalty and the highest confidence is selected as the intermediate result of the standardized inference, the branches with higher penalties are eliminated, and the branches with moderate penalties are downgraded to candidates.

[0083] Preferably, in a clinical setting, for a patient presenting with "chest pain, accompanied by hemoptysis, and a history of hypertension," three branches are generated: the first branch hypothesizes "pulmonary embolism," the second branch hypothesizes "acute myocardial infarction," and the third branch hypothesizes "rib fracture." The symptom-disease logic constraint determines that "rib fractures typically do not directly cause hemoptysis," which is a serious mismatch, thus imposing a high penalty on the third branch. The medical history chronological logic constraint determines that the patient underwent coronary stent implantation six months prior and is taking medication regularly, making the probability of acute myocardial infarction relatively low but not absolutely ruled out, therefore imposing a moderate penalty on the second branch. The medication contraindication constraint checks whether the suggested medication contains any allergens the patient has; if not, there is no penalty. The first branch, with the lowest penalty, is selected as the standard intermediate result, the third branch is filtered out, and the second branch is relegated to a backup. These standard inference intermediate results ensure that invalid inferences that clearly violate clinical common sense have been filtered out before the large model output enters the knowledge graph for verification.

[0084] S42: Perform similarity matching between the intermediate results of the canonical reasoning and the nodes of the structured knowledge graph, and perform logical verification along the causal links in the knowledge graph. After the verification is passed, output the constrained reasoning model.

[0085] In this step, the entities of the intermediate results of the normative reasoning are semantically similar to the nodes of the structured knowledge graph. The associated pairings that are higher than the preset similarity threshold are retained. Logical verification is performed along the symptoms, causes, diseases, examinations and medications and other causal links in the knowledge graph. In this way, reasoning branches with reversed causality, broken links or forced associations are eliminated. After the graph logical constraints are embedded, the verified constraint reasoning results are output.

[0086] S43: Fine-tune the parameters of the constrained reasoning model based on the online incremental learning mechanism and the preset visual interaction feedback data to obtain a reasoning model based on prior knowledge graph constraints.

[0087] In this step, the basic weights of the knowledge graph embedding layer of the constrained reasoning model are frozen based on an online incremental learning mechanism. At the same time, the attention layer and the output fully connected layer are fine-tuned based on the visual interactive feedback data (drag and drop correction of lesion area borders, addition, deletion and modification of symptom labels, confirmation or rejection of abnormal test indicators, and other interactive operations) during the interface annotation process. Meanwhile, the doctor's correction data (including original multimodal features, graph nodes and corrected diagnosis conclusions) are used as a supervision signal input to the preset loss function to iteratively optimize the model's reasoning weights. This is used to reduce the deviation between the model's reasoning and the clinician's actual judgment, and finally obtains a reasoning model that integrates domain prior knowledge and adapts to interactive feedback.

[0088] To clarify the specific method for obtaining intermediate feature analysis results, step S4 includes S44 to S47, specifically:

[0089] S44: Input the multimodal feature set into the inference model, and align and map the domain knowledge graph with the multimodal feature set to obtain multimodal feature data;

[0090] In this step, the multimodal feature set is input into the inference model. Through the entity alignment and attribute mapping logic in the inference model, the visual entities, speech-to-text medical text, target user basic information, and past medical history data in the multimodal feature set are matched and aligned with the standard entity nodes in the domain knowledge graph. Then, through attribute mapping, the preset non-standard attributes are mapped to the attribute set predefined in the graph to eliminate inference interference caused by inconsistent dimensions of multi-source data and non-standard entity naming. The result is multimodal feature data with dimension alignment and entity standardization.

[0091] S45: The multimodal feature data is transformed into knowledge graph nodes, graph entity nodes are mapped through a graph attention network, and feature structure parsing is performed to obtain feature triples;

[0092] In this step, the multimodal feature data is transformed into entity nodes recognizable by the knowledge graph. Then, a graph attention network (GAT) is used to calculate the association attention weights between the entity nodes and standard entity nodes in the domain knowledge graph. Furthermore, based on these attention weights, the optimal matching standard entity node is selected for each entity node, completing the bidirectional mapping between multimodal features and graph entity nodes. The mapping results are then structurally parsed based on the defined "entity-relationship-entity" triple structure in the knowledge graph to generate feature triples.

[0093] S46: Apply a loss penalty to the inference result based on the prior constraint regularization term of the inference model and perform similarity matching to obtain the verified inference result;

[0094] In this step, a joint loss function is constructed based on the built-in prior constraint regularization terms of the inference model (based on the inherent logical rules in domain clinical guidelines, treatment standards, and knowledge graphs). This joint loss function is used to apply a loss penalty to the feature triples and the inference results, removing invalid feature triples that violate common sense in clinical practice. Then, the remaining valid feature triple entities are semantically matched with standard nodes in the domain knowledge graph, filtering out inference branches with entity matching scores below a similarity threshold. Simultaneously, it is verified whether the evidence index associated with each feature triple can be traced back to the original image, audio, or medical record data. Results lacking valid evidence are marked as "low-confidence items to be confirmed." After logical constraints, similarity filtering, and evidence integrity verification, the verified inference results are output.

[0095] The expression for the joint loss function is:

[0096] ;

[0097] In the above formula, For the joint loss function, The prediction loss of the model based on the inference results. For balance coefficient, The loss is due to prior constraints.

[0098] S47: Match the layer markers and feedback nodes of the current stage according to the verification inference results, and perform reverse retrieval of the temporal causal link in combination with the feature triples to construct intermediate feature analysis results.

[0099] In this step, based on the valid feature triples in the verification and inference results, classification markers (highlighted lesion areas, risk level labels) and human-computer interaction feedback nodes (controls requiring doctors to click for confirmation or input supplementary information) in the current augmented reality (AR) layer are matched. Using the current real-time symptom entities (sudden chest pain, shortness of breath, and other symptoms), the relevant entities of the target user's historical medical history, past examination records, and previous treatment plans in the feature triples are retrieved in reverse along the predefined time dimension and causal direction of the domain knowledge graph. A complete causal inference chain covering the current real-time diagnosis and treatment information and the target user's full-cycle medical history is integrated and constructed as the intermediate feature analysis result.

[0100] The expression for the intermediate feature analysis result is:

[0101] ;

[0102] In the above formula, These are the results of intermediate feature analysis. For result identification, Auxiliary analysis content generated for large models. To determine the confidence level of the large model in this outcome, Whether this result needs to be confirmed by a doctor, To generate a larger model version of this result.

[0103] S5: In response to the intermediate feature analysis results, dynamically adjust the annotation display area of ​​the dynamic pose information and generate an enhanced indication signal;

[0104] To clarify the specific method for acquiring the enhanced indication signal, step S5 includes S51 to S54, specifically:

[0105] S51: In response to the intermediate feature analysis results, the coordinate system region of the target user is mapped to the display coordinate system region according to the preset calibration parameters and real-time attitude;

[0106] In this step, in response to the intermediate feature analysis results, based on the pre-calibrated focal length, principal point coordinates, distortion coefficient parameters and real-time tracked pose data (rotation matrix and translation vector) of the augmented reality (AR) terminal, the coordinate system region of the target user is mapped from the image coordinate system (pixel coordinates) to the three-dimensional display coordinate system in the augmented reality (AR) terminal space through perspective projection transformation, thus obtaining the display coordinate system region.

[0107] The expression for the display coordinate system region is:

[0108] ;

[0109] In the above formula, The display coordinate system area for the target user. For coordinate mapping operations, The current terminal posture, Pre-calibration parameters for augmented reality terminals.

[0110] S52: Expand the boundary of the display coordinate system area based on the preset display safety spacing to generate a display protection area;

[0111] In this step, the value of the display safety distance is determined based on the observation requirements of the diagnostic and treatment scenario. For detailed observation areas such as the face, tongue, and lesions on the body surface, the display safety distance is set to 15% to 25% of the area width to ensure that auxiliary annotation information does not obscure key observation areas. For larger areas such as the torso, the display safety distance is set to a fixed value of 3 to 8 centimeters in the display coordinate system. The display protection area is used to cover the target user's body area and the surrounding buffer zone, serving as an absolutely prohibited annotation area for auxiliary information. No text labels, indicator pop-ups, or graphic markers may geometrically overlap with this area.

[0112] The expression for the display protection area is:

[0113] ;

[0114] In the above formula, Display protected area, Expand the display area of ​​the target user's device. The display coordinate system area for the target user. This indicates the outer distance of the safety clearance.

[0115] S53: Identify the already displayed content and potential obstruction areas in the display protection area, and select blank areas as candidate display areas;

[0116] In this step, overlapping spatial blocks are filtered out from the display protection area. Simultaneously, rendered historical annotation text, indicator pop-ups, prompt charts, and other deployed display content within the current field of view are identified and constructed as occupied space. Potentially obstructing areas that overlap with the occupied space, exceed the terminal display boundary, have severe corner distortion, or partially fall within the display protection area are eliminated. Blank blocks that do not conflict with any existing annotations, are completely within the effective field of view, and completely avoid the display protection area are grouped together as candidate display areas.

[0117] S54: The region in the candidate display area that is farthest from the boundary of the key part of the target user and has the smallest positional change is used as the marked display area, and an enhanced indication signal is generated.

[0118] In this step, the labeled display area meets multiple constraints, including being furthest from the boundary of the key observation part farthest from the target user, having a smooth change in position between frames without large jumps, and being far from the screen edge. Then, the labeled display area, the consultation identification information, and the timestamp information are structurally encapsulated to generate an enhanced indication signal carrying the session identifier and the target user's target identifier.

[0119] The labeled display area is:

[0120] ;

[0121] In the above formula, To mark the display area, For the first One candidate display area These are the weighting coefficients. The distance between the candidate region and the target user's display area. This represents the change in the candidate region's display position relative to the previous time step. The degree to which the candidate region approaches the boundary of the display field of view.

[0122] S6: Based on the enhanced indication signal and the preset real-time feedback data, perform closed-loop correction on the intermediate feature analysis results and output visualized rendering display data.

[0123] To clarify the specific method for obtaining data for visualization rendering, step S6 includes S61 to S64, specifically:

[0124] S61: Based on the enhanced indication signal, call the target user positioning data and inertial attitude data at the current moment as real-time feedback data;

[0125] In this step, the enhanced indication signal is parsed, the unified timestamp carried in the enhanced indication signal is extracted, and the unified timestamp is used as the retrieval index to retrieve the target user positioning data and the inertial attitude data at the current time for time alignment and combination, forming real-time feedback data for real-time correction of the marked position.

[0126] S62: Based on the running status update of the enhanced indication signal and the intermediate feature analysis results, the updated parameters are obtained;

[0127] In this step, based on the layer markers (risk label layer, anatomical structure layer, and history layer), marker priority, and visualization display type (text box, pop-up window, highlighted circle, and arrow guide) corresponding to the marked display area in the enhanced indication signal, and combined with the inference confidence and the running status of the markers awaiting doctor confirmation stored in the intermediate feature analysis results, the rendering status field is updated to generate update parameters including display priority, pop-up window size, text font size, highlighted marker on / off, and evidence folding status, which are used to control the display format and interaction permissions of the information.

[0128] S63: Dynamically correct the candidate display area of ​​the intermediate feature analysis result in the augmented reality display coordinate system based on the real-time feedback data to obtain the corrected display area parameters;

[0129] In this step, the candidate display area of ​​the intermediate feature analysis result in the display coordinate system is dynamically corrected based on the positioning data and attitude data in the real-time feedback data. The specific triggering conditions are as follows:

[0130] When the target user moves or the doctor turns his head, causing a change in the terminal's posture, the coordinate mapping, protection area expansion, and candidate area screening processes in steps S51 to S53 are re-executed. At the same time, due to the problem of flickering of the annotation screen caused by slight jitter in the tracking system, the annotation display area of ​​the previous frame is subjected to inter-frame smoothing interpolation processing to solve the problem of flickering of the annotation screen.

[0131] When there is a large area of ​​occlusion of the target user, insufficient effective matching feature points, or the location confidence is lower than the preset number of feature points, the screening range of the candidate display area is expanded outward, and the candidate area is re-screened within the expanded display area. Finally, the corrected display area parameters are output. The corrected display area parameters include the area coordinate boundary, the maximum allowable displacement threshold, and the temporary occlusion avoidance offset.

[0132] S64: Establish a data association based on the updated parameters and the corrected display area parameters, and output the visualization rendering display data.

[0133] In this step, the updated parameters, the corrected display area parameters, the intermediate feature analysis results, the session identifier, the patient identifier, and the time-series timestamp are bound together to establish an association index and construct a rendering data package. Subsequently, the rendering data package is input into the graphics rendering unit (GPU) of the augmented reality (AR) terminal for real-time overlay display, outputting visual rendering display data.

[0134] The visualization rendering data and multimodal feature set are synchronously uploaded to the server and stored in a closed-loop update dataset to provide original human-computer interaction and visualization process data support for subsequent manual correction, offline model parameter optimization, and annotation strategy adjustment.

[0135] Example 2:

[0136] This embodiment provides an augmented reality multimodal information processing system for diagnostic and treatment scenarios, the system comprising:

[0137] The interaction module is used to generate consultation identification information associated with the target user in response to interactive triggering operations;

[0138] The pose module is used to determine the dynamic pose information of the target user in the monitoring space based on the consultation identification information and the multi-source perception data of the collected users.

[0139] The alignment module is used to align the dynamic pose information with the pre-stored time-series recording data in the spatiotemporal dimension to construct a multimodal feature set;

[0140] The reasoning module is used to input the multimodal feature set into a reasoning model based on prior knowledge graph constraints for feature reasoning and generate intermediate feature analysis results for interface annotation.

[0141] To clarify the specific methods for obtaining the reasoning model in the reasoning module, the following are included:

[0142] The mapping unit is used to perform entity alignment and attribute mapping on the multimodal feature set based on a preset domain knowledge graph to obtain structured knowledge graph nodes.

[0143] The penalty unit is used to apply loss penalties to inference results that do not meet the preset constraints based on the prior constraint regularization term, so as to obtain the intermediate results of the canonical inference.

[0144] The matching unit is used to perform similarity matching between the intermediate results of the canonical reasoning and the nodes of the structured knowledge graph, and to perform logical verification along the causal links in the knowledge graph. After the verification is passed, the constrained reasoning model is output.

[0145] The fine-tuning unit is used to fine-tune the parameters of the constrained reasoning model based on the online incremental learning mechanism and preset visual interaction feedback data, so as to obtain a reasoning model based on prior knowledge graph constraints.

[0146] An adjustment module is used to dynamically adjust the annotation display area of ​​the dynamic pose information in response to the intermediate feature analysis results, and generate an enhanced indication signal.

[0147] To clarify the specific methods for obtaining the adjustment module, the following are included:

[0148] The display mapping unit is used to respond to the intermediate feature analysis results and map the target user's coordinate system area to a display coordinate system area according to preset calibration parameters and real-time attitude.

[0149] The extension unit is used to extend the boundary of the display coordinate system area based on a preset display safety spacing to generate a display protection area;

[0150] The identification unit is used to identify the displayed content and potential obstruction areas in the display protection area and select blank areas as candidate display areas.

[0151] The annotation display unit is used to select the region in the candidate display region that is farthest from the boundary of the key part of the target user and has the smallest positional change as the annotation display region, and to generate an enhanced indication signal.

[0152] The correction module is used to perform closed-loop correction on the intermediate feature analysis results based on the enhanced indication signal and preset real-time feedback data, and output visualized rendering display data.

[0153] To clarify the specific methods for obtaining the correction module, the following are provided:

[0154] The calling unit is used to call the target user's positioning data and inertial attitude data at the current moment according to the enhanced indication signal, as real-time feedback data;

[0155] An update unit is used to update the running status based on the enhanced indication signal and the intermediate feature analysis results to obtain update parameters;

[0156] The correction unit is used to dynamically correct the candidate display area of ​​the intermediate feature analysis result in the augmented reality display coordinate system according to the real-time feedback data, and obtain the corrected display area parameters.

[0157] The association establishment unit is used to establish data associations based on the updated parameters and the corrected display area parameters, and output visualization rendering display data.

[0158] It should be noted that the specific methods by which each module performs operations in the system described in the above embodiments have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0159] Example 3:

[0160] Corresponding to the above method embodiments, this embodiment also provides an augmented reality multimodal information processing device for diagnosis and treatment scenarios. The augmented reality multimodal information processing device for diagnosis and treatment scenarios described below and the augmented reality multimodal information processing method for diagnosis and treatment scenarios described above can be referred to in correspondence with each other.

[0161] Figure 2 This is a block diagram illustrating an augmented reality multimodal information processing device 800 for a medical scenario, according to an exemplary embodiment. Figure 2 As shown, the augmented reality multimodal information processing device 800 for medical scenarios may include: a processor 801 and a memory 802. The augmented reality multimodal information processing device 800 may also include one or more of a multimedia component 803, an I / O interface 804, and a communication component 805.

[0162] The processor 801 controls the overall operation of the augmented reality multimodal information processing device 800 for medical scenarios to complete all or part of the steps in the aforementioned augmented reality multimodal information processing method for medical scenarios. The memory 802 stores various types of data to support the operation of the augmented reality multimodal information processing device 800 for medical scenarios. This data may include, for example, instructions for any application or method operating on the augmented reality multimodal information processing device 800 for medical scenarios, as well as application-related data, such as contact data, sent and received messages, images, audio, video, etc. The memory 802 can be implemented using any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The multimedia component 803 may include a screen and an audio component. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in the memory 802 or transmitted via the communication component 805. The audio component also includes at least one speaker for outputting audio signals. I / O interface 804 provides an interface between processor 801 and other interface modules, such as keyboards, mice, and buttons. These buttons can be virtual or physical. Communication component 805 is used for wired or wireless communication between the augmented reality multimodal information processing device 800 for diagnostic scenarios and other devices. Wireless communication includes Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, or 4G, or a combination thereof. Therefore, the corresponding communication component 805 may include a Wi-Fi module, a Bluetooth module, and an NFC module.

[0163] In an exemplary embodiment, the augmented reality multimodal information processing device 800 for diagnosis and treatment scenarios can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to execute the above-described augmented reality multimodal information processing method for diagnosis and treatment scenarios.

[0164] Example 4:

[0165] Corresponding to the above method embodiments, this embodiment also provides a medium. The medium described below can be referred to in conjunction with the augmented reality multimodal information processing method for diagnosis and treatment scenarios described above.

[0166] A medium storing a computer program, which, when executed by a processor, implements the steps of the augmented reality multimodal information processing method for diagnostic and treatment scenarios described in the above method embodiments.

[0167] The medium can specifically be any medium capable of storing program code, such as a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0168] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0169] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An augmented reality multimodal information processing method for medical scenarios, characterized in that, include: In response to interactive triggering operations, generate consultation identification information associated with the target user; Based on the consultation identification information and the multi-source perception data of the collected users, the dynamic pose information of the target user in the monitoring space is determined. The dynamic pose information is aligned with the pre-stored time-series recording data in the spatiotemporal dimension to construct a multimodal feature set; The multimodal feature set is input into a reasoning model constrained by prior knowledge graphs for feature reasoning, generating intermediate feature analysis results for interface annotation. In response to the intermediate feature analysis results, the annotation display area of ​​the dynamic pose information is dynamically adjusted to generate an enhanced indication signal; The intermediate feature analysis results are corrected in a closed loop based on the enhanced indication signal and preset real-time feedback data, and the data is then output as a visual rendering display.

2. The augmented reality multimodal information processing method for diagnostic and treatment scenarios according to claim 1, characterized in that, The reasoning model based on prior knowledge graph constraints includes: Based on a preset domain knowledge graph, entity alignment and attribute mapping are performed on the multimodal feature set to obtain structured knowledge graph nodes; Based on the prior constraint regularization term, a loss penalty is imposed on the reasoning result that does not meet the preset constraint conditions to obtain the intermediate result of the canonical reasoning; The intermediate results of the normative reasoning are matched with the nodes of the structured knowledge graph for similarity, and logical verification is performed along the causal links in the knowledge graph. After the verification is passed, the constrained reasoning model is output. The parameters of the constrained reasoning model are fine-tuned based on the online incremental learning mechanism and the preset visual interaction feedback data to obtain a reasoning model based on prior knowledge graph constraints.

3. The augmented reality multimodal information processing method for diagnostic and treatment scenarios according to claim 2, characterized in that, The multimodal feature set is input into a reasoning model constrained by a prior knowledge graph for feature reasoning, generating intermediate feature analysis results for interface annotation, including: The multimodal feature set is input into the inference model, and the domain knowledge graph is aligned with the multimodal feature set and its attributes are mapped to obtain multimodal feature data. The multimodal feature data is transformed into knowledge graph nodes, graph entity nodes are mapped through a graph attention network, and feature structure parsing is performed to obtain feature triples. Based on the prior constraint regularization term of the inference model, a loss penalty is applied to the inference result and similarity matching is performed to obtain the verified inference result; Based on the verification and inference results, the layer markers and feedback nodes of the current stage are matched, and the temporal causal link is reversed by combining the feature triples to construct intermediate feature analysis results.

4. The augmented reality multimodal information processing method for diagnostic and treatment scenarios according to claim 1, characterized in that, Based on the consultation identification information and the multi-source sensing data of the collected users, the dynamic pose information of the target user in the monitoring space is determined, including: Based on the consultation identification information, time-series images are retrieved, and visual data of the target user in the time-series images are collected to obtain multi-source perception data. Local feature points are extracted from the multi-source sensing data, and valid matching points are filtered by matching the feature points of the current frame with the feature points of the previous frame. Based on the effective matching points, the pose transformation within the time-series image is calculated, and the pose state is estimated by combining the preset inertial sensing data to obtain the actual position change of the target user. The pose estimation of the current frame is updated based on the actual position change. When the number of valid matching points is lower than the preset matching threshold, the person detection is re-executed to generate real-time spatial positioning and posture data of the user as dynamic pose information.

5. The augmented reality multimodal information processing method for diagnostic and treatment scenarios according to claim 1, characterized in that, In response to the intermediate feature analysis results, the annotation display area of ​​the dynamic pose information is dynamically adjusted to generate an enhanced indication signal, including: In response to the intermediate feature analysis results, the coordinate system region of the target user is mapped to the display coordinate system region according to the preset calibration parameters and real-time attitude; Based on a preset display safety spacing, the boundary of the display coordinate system area is expanded to generate a display protection area; The displayed content and potential obstruction areas in the display protection area are identified, and blank areas are selected as candidate display areas. The region that is furthest from the boundary of the target user's key part and has the smallest positional change in the candidate display region is used as the marked display region, and an enhanced indication signal is generated.

6. The augmented reality multimodal information processing method for diagnostic and treatment scenarios according to claim 1, characterized in that, The intermediate feature analysis results are corrected in a closed loop based on the enhanced indication signal and preset real-time feedback data, and the resulting visualization rendering data is output, including: The target user's positioning data and inertial attitude data at the current moment are retrieved based on the enhanced indication signal as real-time feedback data; Update parameters are obtained by updating the running status based on the enhanced indication signal and the intermediate feature analysis results; Based on the real-time feedback data, the candidate display area of ​​the intermediate feature analysis result in the augmented reality display coordinate system is dynamically corrected to obtain the corrected display area parameters; Establish data association based on the updated parameters and the corrected display area parameters, and output visual rendering display data.

7. An augmented reality multimodal information processing system for medical scenarios, characterized in that, include: The interaction module is used to generate consultation identification information associated with the target user in response to interactive triggering operations; The pose module is used to determine the dynamic pose information of the target user in the monitoring space based on the consultation identification information and the multi-source perception data of the collected users. The alignment module is used to align the dynamic pose information with the pre-stored time-series recording data in the spatiotemporal dimension to construct a multimodal feature set; The reasoning module is used to input the multimodal feature set into a reasoning model based on prior knowledge graph constraints for feature reasoning and generate intermediate feature analysis results for interface annotation. An adjustment module is used to dynamically adjust the annotation display area of ​​the dynamic pose information in response to the intermediate feature analysis results, and generate an enhanced indication signal. The correction module is used to perform closed-loop correction on the intermediate feature analysis results based on the enhanced indication signal and preset real-time feedback data, and output visualized rendering display data.

8. The augmented reality multimodal information processing system for diagnostic and treatment scenarios according to claim 7, characterized in that, The reasoning model of the reasoning module includes: The mapping unit is used to perform entity alignment and attribute mapping on the multimodal feature set based on a preset domain knowledge graph to obtain structured knowledge graph nodes. The penalty unit is used to apply loss penalties to inference results that do not meet the preset constraints based on the prior constraint regularization term, so as to obtain the intermediate results of the canonical inference. The matching unit is used to perform similarity matching between the intermediate results of the canonical reasoning and the nodes of the structured knowledge graph, and to perform logical verification along the causal links in the knowledge graph. After the verification is passed, the constrained reasoning model is output. The fine-tuning unit is used to fine-tune the parameters of the constrained reasoning model based on the online incremental learning mechanism and preset visual interaction feedback data, so as to obtain a reasoning model based on prior knowledge graph constraints.

9. The augmented reality multimodal information processing system for diagnostic and treatment scenarios according to claim 7, characterized in that, The adjustment module includes: The display mapping unit is used to respond to the intermediate feature analysis results and map the target user's coordinate system area to a display coordinate system area according to preset calibration parameters and real-time attitude. The extension unit is used to extend the boundary of the display coordinate system area based on a preset display safety spacing to generate a display protection area; The identification unit is used to identify the displayed content and potential obstruction areas in the display protection area and select blank areas as candidate display areas. The annotation display unit is used to select the region in the candidate display region that is farthest from the boundary of the key part of the target user and has the smallest positional change as the annotation display region, and to generate an enhanced indication signal.

10. The augmented reality multimodal information processing system for diagnostic and treatment scenarios according to claim 7, characterized in that, The correction module includes: The calling unit is used to call the target user's positioning data and inertial attitude data at the current moment according to the enhanced indication signal, as real-time feedback data; An update unit is used to update the running status based on the enhanced indication signal and the intermediate feature analysis results to obtain update parameters; The correction unit is used to dynamically correct the candidate display area of ​​the intermediate feature analysis result in the augmented reality display coordinate system according to the real-time feedback data, and obtain the corrected display area parameters. The association establishment unit is used to establish data associations based on the updated parameters and the corrected display area parameters, and output visualization rendering display data.

Images