Medical care decision training method and system based on large model and augmented reality

Abstract

The application belongs to the field of virtual simulation training, and discloses a medical and health care decision training method and system based on a large model and augmented reality. t , a voice signal s t issued by a nurse, and an interaction event e t to generate a structured semantic behavior vector a t ; in step two, the a t , a state vector S t of a current patient, and a patient individual parameter vector c are input into a state generation main model to obtain a new state vector S t+1 of the patient; in step three, state feedback masks of various state dimensions are generated based on the S t and the S t+1 , and a rationality score r t of a current behavior of the nurse is calculated; in step four, a priority score of a candidate task is calculated, and a task τ1 t+1 for a next round of training is determined based on the priority score; in step five, a structured semantic behavior vector a t+1 for the next round of training is obtained based on the τ1 t+1 . The application enhances the decision-making ability and operation agility of a nurse in a complex continuous nursing scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of virtual simulation training, and in particular to a medical and health care decision-making training method and system based on large models and augmented reality. BACKGROUND

[0002] With the continuous complication and diversification of the clinical medical environment, nursing staff not only need to master basic operation skills in actual work, but also need to have good emergency handling ability, decision-making ability and interpersonal communication ability. However, the current nursing training methods generally have the problem of "theoretical operation light, one-way infusion teaching", and lack the restoration of real clinical scenes and the training of decision-making ability. In the traditional nursing teaching mode, paper teaching materials, standardized patient cases or physical simulation dolls are often relied on, and students often train in static, pre-set and single situations. This kind of way is difficult to simulate the real-time changes of patient state and the sudden conditions in the clinic, and it is also impossible to reflect the adaptability and judgment ability of nurses in the continuous decision-making process of complex disease development. In addition, some higher vocational and medical colleges have also introduced virtual simulation or VR / AR systems, but these systems mostly use script type and linear process control, and the changes of patient state are triggered by the fixed process preset by the system, lacking real-time perception and feedback mechanism for the behavior of nurses. This means that whether the operation of the nurse is appropriate will not really trigger the reaction of the patient, and the nurse cannot form the causal cognition of his own decision-making by observing the "real changes of patient state" in the training process. In addition, the existing systems generally lack the real simulation of "operation error" scenes, and students can rarely see the chain effect of errors on patient state even if they make mistakes in the operation process, not to mention the formation of the ability of "error remediation and correction" through training.

[0003] Meanwhile, the rapid development of artificial intelligence in recent years, especially large language models (such as GPT and DeepSeek), has made it possible to build training systems with dynamic responses. These models possess powerful multi-turn reasoning and semantic understanding capabilities, enabling them to generate personalized and continuous responses in complex contexts. If large models can be integrated with clinical nursing knowledge, it may be possible to construct a "virtual patient" that infers changes in patient status in real time based on nurses' behavior; combining this with augmented reality (AR) technology to present these dynamic changes will make the training scenario more immersive and clinically realistic. However, existing training systems generally do not apply large models to the generation of dynamic patient conditions, and lack a mechanism for designing a closed loop between nurse behavior, model feedback, and AR presentation. This gap makes it difficult for current nursing training to achieve the closed-loop training goal of "behavior-driven feedback, error-induced consequences, and system-assisted error correction." Therefore, there is an urgent need for a collaborative training system based on large models and AR that can support real-time evolution simulation of complex patient conditions, behavior-driven state feedback mechanisms, and personalized decision-making path training frameworks, thereby better serving the training needs of modern nursing professionals. Summary of the Invention

[0004] The purpose of this invention is to disclose a method and system for training medical and health care decisions based on large models and augmented reality, so as to solve the technical problems pointed out in the background art.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] On the one hand, this invention provides a training method for healthcare decision-making based on large models and augmented reality, including:

[0007] Step 1: After the nurse enters the training scenario, collect the video image frame sequence v captured by the AR headset. t The voice signals issued by the nurse t and recorded interaction events e t Based on v t s t and e t Generate structured semantic behavior vector a t ;

[0008] Step 2: Obtain the current patient's state vector S t The patient's individual parameter vector c, S t c and a t Input to the state generation master model to obtain the patient's new state vector S t+1 ;

[0009] Step 3, based on S t and S t+1generate a state feedback mask for each state dimension, calculate a rationality score r of the current behavior of the nurse based on the state feedback mask t ;

[0010] Step four, calculate a priority score of the candidate task based on the state feedback mask, determine the task τ1 for the next round of training based on the priority score t+1 ;

[0011] Step five, based on τ1 t+1 , obtain the structured semantic behavior vector a for the next round of training t+1 .

[0012] Further, step one includes:

[0013] Concatenate v t , s t and e t into a unified dimensional feature representation x t ;

[0014] Use the structured semantic encoding module NurseActionSC to perform context aggregation on the sequence of unified dimensional feature representations {x t-2 , x t-1 , x t} of the last three time points, and output the structured semantic behavior vector a t .

[0015] Further, S t includes basic vital signs, functional states and environmental dependent variables, and the individual parameter vector c includes age, gender, allergy history and chronic disease labels.

[0016] Further, the calculation formula of S t+1 is:

[0017] S t+1 = f θ (S t , a t , c) + λ·G(S t , a t )

[0018] S t+1 is the state vector of a new patient, representing the virtual patient's response after the nurse's operation; f θ (·) is a state generation main model, which performs multi-dimensional variable prediction based on a Transformer structure; G(S t , a t ) is a causal regulation function for state rearrangement correction according to the difference between the medical causal graph and the current state; λ is a coefficient for controlling the causal enhancement term.

[0019] Further, the calculation formula of G(S t ,a t ) is:

[0020] G(S t ,a t ) = ReLU(W c ·concat(S t ,a t ))

[0021] W c is a causal regulation weight matrix, which is spliced by the state vector and the structured semantic behavior vector; concat(S t ,a t ) represents splicing the current state vector and the structured semantic behavior vector; ReLU(·) is an activation function, which is used to limit the output to express only upward adjustment trend, to prevent unreasonable reverse adjustment; d is the dimension of a t .

[0022] Further, the calculation process of the state feedback mask of the i-th state dimension includes:

[0023]

[0024] σ(·) is a threshold activation function;

[0025] and represent the i-th state dimension in S t+1 and S t , respectively;

[0026] α i is the clinical risk weighting coefficient of the i-th state dimension;

[0027] τ1 i is a set dimension-sensitive threshold value;

[0028] n represents the dimension of the state vector of the patient.

[0029] Further, step four includes:

[0030] The candidate task library represents M is the total number of candidate tasks;

[0031] The calculation formula of the priority score of the candidate task τ i is:

[0032]

[0033] ψ(τ i ) is the candidate task τ ithe adaptation priority score of the jth state dimension;

[0034] a state feedback mask representing the jth state dimension;

[0035] β j an importance weight of the jth state dimension, determined by the medical atlas;

[0036] if the jth state dimension has appeared in the error-prone state indicator record ε more than a set threshold, 1, otherwise 0;

[0037] λ is an incentive coefficient for matching the error dimension;

[0038] ρ is a coefficient for punishing repeated training;

[0039] Redundancy(·) represents the similarity of the candidate task τ i to the historical training tasks ;

[0040] The candidate task with the highest priority score is selected from all candidate tasks as the task τ1 t+1 for the next round of training.

[0041] Further, step five includes:

[0042] τ1 t+1 contains:

[0043] an initial state vector configured for the patient's basic state at the beginning of the new task;

[0044] a list of operation targets containing predefined training targets;

[0045] a behavior type label for task definition;

[0046] a context parameter field for task interaction loading;

[0047] According to , the virtual patient state configuration is started, and the corresponding vital sign simulation module is activated;

[0048] synchronize the list of operation targets and the context parameter field in τ1 t+1 to the nurse interaction layer;

[0049] After the nurse performs the next round of training τ1 t+1 , record the video image frame sequence v t+1 , the voice signal s t+1 and the interaction event e t+1 issued by the nurse during the operation;

[0050] v t+1 , s t+1 , and e t+1 are spliced into a unified dimensional feature representation x t+1 for the next round of training;

[0051] Generate a new round of structured semantic behavior vector a t+1 :

[0052] a t+1 = NurseActionSC(x t+1 )

[0053] a t+1 is the direct input of state reasoning in the subsequent step.

[0054] On the other hand, the present application provides a medical and health care decision-making training system based on large models and augmented reality, comprising a generation module, a first acquisition module, a first calculation module, a second calculation module and a second acquisition module;

[0055] The generation module is used to collect the video image frame sequence v t , the voice signal s t issued by the nurse and the recorded interaction event e t collected by the AR headset after the nurse enters the training scene, generate a structured semantic behavior vector a t based on v t , s t and e t ;

[0056] The first acquisition module is used to acquire the state vector S t of the current patient and the individual parameter vector c of the patient, and input S t , c and a t to the state generation master model to obtain a new state vector S t+1 of the patient;

[0057] The first calculation module is used to generate a state feedback mask for each state dimension based on S t and S t+1 , and calculate a rationality score r t of the current behavior of the nurse based on the state feedback mask;

[0058] The second calculation module is used to calculate a priority score of the candidate task based on the state feedback mask, and determine a task τ1 t+1 for the next round of training based on the priority score;

[0059] The second acquisition module is used to acquire a structured semantic behavior vector a t+1 for the next round of training based on τ1 t+1 .

[0060] Beneficial effects:

[0061] In view of the problems of patient state staticization, lack of real-time feedback, inability to cause consequences for errors, and rigid path training in existing nursing training systems, the present application proposes a nursing decision training method and system that combines large model reasoning and augmented reality feedback mechanism. First, by introducing a large language model with medical knowledge understanding ability, the present application can dynamically reason the state changes of virtual patients according to the operation behavior of nurses in the training scene, overcoming the problems of fixed illness changes and lack of response in traditional systems. Second, by integrating with AR technology, the system can realistically present the patient's physical sign evolution process on the visual level, such as skin color, consciousness state, vital signs, etc., realizing the interactive closed loop between nurse behavior and patient feedback, and improving the immersion and causal cognition ability of training. Further, the present application also introduces a simulation mechanism and emergency feedback design for nursing errors, so that nurses can observe the specific illness changes caused by errors after operation mistakes and conduct multiple rounds of remedial operation training, solving the one-sided problem of "error as failure, unable to correct" in existing training. Finally, the system can dynamically adjust the training path combined with the training history and current performance of nurses, realizing personalized, multi-round, adaptive training mode, and enhancing the decision-making ability and operation agility of nurses in complex continuous nursing scenarios. The above innovations have broken through the existing nursing training technology in terms of systematicness, interactivity and intelligence, and have significant practical application value and engineering implementation possibility. BRIEF DESCRIPTION OF DRAWINGS

[0062] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0063] Figure 1 Fig. 1 is a schematic diagram of the medical and health care nursing decision training method based on large model and augmented reality of the present application. DETAILED DESCRIPTION

[0064] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work belong to the scope of protection of the present application.

[0065] As Figure 1 shown in an embodiment, the present application provides a medical and health care decision-making training method based on a large model and augmented reality, comprising:

[0066] Step one, after the nurse enters the training scene, collect the video image frame sequence v t collected by the AR head-mounted display, the voice signal s t issued by the nurse, and the recorded interaction event e t , generate a structured semantic behavior vector a t based on v t , s t and e t ;

[0067] v t contains hand actions, body movements, etc.; s t contains "preparing infusion", "please turn around", etc.; e t contains instrument operation, menu click, etc., and the system of the AR head-mounted display records; the three types of data are synchronously labeled with time stamps and directly written into the multi-modal perception cache area of the system through a pre-defined interface;

[0068] Step two, obtain the state vector S t of the current patient and the individual parameter vector c of the patient, input S t , c and a t into a state generation main model to obtain a new state vector S t+1 of the patient;

[0069] Step three, generate a state feedback mask for each state dimension based on S t and S t+1 , and calculate a rationality score r t of the current behavior of the nurse based on the state feedback mask;

[0070] Step four, calculate a priority score of the candidate task based on the state feedback mask, and determine a task τ1 t+1 for the next round of training based on the priority score;

[0071] Step 5, based on τ1 t+1 Obtain the structured semantic behavior vector a for the next round of training t+1 .

[0072] Furthermore, step one includes:

[0073] After the nurse enters the training scenario, the system activates the multimodal acquisition module, collecting current behavioral information in 200ms increments to form a time slice d. t ={v t ,s t ,e t};

[0074] v t s t and e t Concatenated into a feature representation x of a unified dimension t ;

[0075] x t =φ v (v t )||φ s (s t )||φ e (e t )

[0076] φ represents the feature representation of the current time with uniform dimension. v (v t ) is a visual encoding function, and its input is a sequence of video image frames v. t Output 128 dimensions;

[0077] φ s (s t ) is a speech encoding function, and the input is a speech signal s. t Output 64 dimensions;

[0078] φ e (e t ) is the interaction event encoding function, and the input is the interaction event e recorded by the system. t Output 16 dimensions;

[0079] || represents the vector concatenation operation.

[0080] The structured semantic coding module NurseActionSC is used to process the feature representation sequence {x} of the most recent three time steps with a unified dimension. t-2 ,x t-1 ,x t Perform context aggregation and output a structured semantic behavior vector a. t .

[0081] a t = NurseActionSC(x t-2 ,x t-1 ,x t )

[0082] is the structured semantic action vector, as the large model inference input;

[0083] NurseActionSC(·) is the structured semantic encoding module, which uses attention mechanism internally;

[0084] x t-2 ,x t-1 ,x t are the unified dimension feature representations of the last three time points, which are used to capture the context action intention.

[0085] This module captures the relationship between action patterns (such as moving first and then injecting), language intentions (such as confirming the condition and then prompting medication), and operation objects through self-attention mechanism, forming an encoding result with high semantic expression. The final a t is a 256-dimensional vector, which contains four sub-structure fields: action type (such as "injection"), object area (such as "right upper arm"), additional language intention (such as "calm down"), and system confidence score (such as 0.87). This structured semantic action vector will be used as the main input of the subsequent state inference model f θ (S t ,a t ).

[0086] Compared with general behavior recognition methods, this scheme has three advantages designed specifically for the nursing scene. First, the system-level data of the interaction event e t is retained, so that the nurse's operation is not limited to visible actions, but also includes "device configuration" and other key contexts, avoiding missing implicit nursing behaviors. Second, the speech processing module not only transcribes text, but also models language states by combining pronunciation rhythm and semantic relationship, which is particularly suitable for evaluating the nurse's communication behavior, such as whether the patient is properly comforted or risk is reported. Third, the structured semantic action vector a t is not simply a classification result, but a multi-field and multi-granularity representation, ensuring that the subsequent inference model can be used not only for state prediction, but also for behavior evaluation and task scheduling. During system operation, a t is continuously input to the subsequent state evolution model in time sequence and serves as an index marker for each round of training data. Since a tWith the structural hierarchy, it can also be used for "behavior matching and weak link identification" in the path scheduling stage. The entire coding scheme not only solves the problem of "unclear operation behavior and various data types", but also builds a unified behavior expression portal required by the subsequent system, laying a foundation for realizing the nursing training decision reasoning driven by large models.

[0087] Further, S t including basic vital signs (such as heart rate, blood pressure, body temperature), functional status (such as consciousness level, pain index, respiratory rate) and environment-dependent variables (such as current bed position, infusion status, etc.), S t The general dimension n∈[12,20]; the individual parameter vector c includes age, gender, allergy history and chronic disease label.

[0088] The individual parameter vector can make the state evolution have individual differences.

[0089] Step two aims to generate a structured semantic behavior vector a t and the state vector S t of the current patient, and then infer the new state vector S θ of the patient based on the state evolution model f t+1 , simulate the real impact of the nurse's behavior on the patient's physiological state and symptoms in the current situation. This module is the "core engine" of the invention, responsible for breaking the causal chain between "nursing operations" and "patient reactions", and establishing a high-fidelity interactive relationship between the behavior decision of the nurse and the feedback of the patient's state. Compared with the static and script-based patient reaction in traditional simulation training, this scheme can generate real-time multivariate state evolution and support cross-time evolution and operation chain effect, which is the key point of the invention that distinguishes it from the prior art.

[0090] Further, S t+1 The calculation formula is:

[0091] S t+1 =f θ (S t ,a t ,c)+λ·G(S t ,a t )

[0092] S t+1 is the new state vector of the patient, representing the virtual patient reaction after the nurse's operation; f θ (·) is the state generation main model, which is based on the Transformer structure for multivariate prediction; G(S t ,a t ) is the causal regulation function of S t , used to rearrange and correct the state according to the medical causal graph and the difference between the current state; λ is the coefficient controlling the causal enhancement term.

[0093] λ generally ranges from 0.1 to 1.0, and a common method is to set a small value (such as 0.2 or 0.3) at the beginning of training to ensure that the main model dominates the training, and then fine-tune according to the prediction accuracy of the causal related variables (such as blood pressure, respiratory rate, etc.) on the validation set, gradually increase to 0.5 or higher, and finally select the value corresponding to the minimum error of the key indicators and the optimal overall prediction stability.

[0094] f θ (·) is jointly driven by a large language model embedding and a medical state atlas. The basic process is: the state generation main model f θ takes S t , a t and c as input to generate a new state vector S t+1 under the structure control network. The modeling method not only depends on data fitting, but also integrates the logical causal weights between medical variables, ensuring that the output changes have physiological reasonableness and clinical credibility.

[0095] Further, the calculation formula of G(S t ,a t ) is:

[0096] G(S t ,a t ) = ReLU(W c ·concat(S t ,a t ))

[0097] W c is a causal regulation weight matrix, which is spliced from the state vector and the structured semantic behavior vector. When splicing, each dimension of the state vector and each dimension of the structured semantic behavior vector are spliced in the column direction to form a new input vector, and the length of the new input vector is equal to the dimension of the state vector plus the dimension of the structured semantic behavior vector.

[0098] concat(S t ,a t ) represents splicing the current state vector and the structured semantic behavior vector; ReLU(·) is an activation function, which is used to limit the output to only express upward adjustment trend, to prevent unreasonable reverse adjustment; d is the dimension of a t , which is kept at 256. The output G(S t ,a t ) is an n-dimensional vector, which only adjusts part of the dimensions (such as heart rate ↑, body temperature no change, consciousness ↓).

[0099] The causal regulation function G(·) is the key innovation of this step, which mainly introduces an asymmetric and learnable variation adjustment mechanism to control the direction and strength of the "chain effect" in the disease reasoning result. Specifically, for some structured semantic behavior vectors a t If it triggers a variable with a high-sensitive causal relationship (e.g., large dose of analgesia → respiratory depression), the system will introduce a regulation matrix W c in G(·) to emphasize the impact dimension and make state corrections to simulate "non-local pathological response". This mechanism not only enhances the clinical consistency of simulation, but also enables nurses to more intuitively understand the "general consequences" of their behaviors through state feedback.

[0100] For example, if the current patient's state vector S t has a heart rate of 98 and a body temperature of 37.1°C, and the nurse performs a rapid infusion behavior (which is the high-confidence operation action "bolus infusion" in a t , the system will activate the causal weight edges related to blood pressure and heart rate, and adjust the "blood pressure" and "heart rate" dimensions in S t+1 in G(·).

[0101] To prevent the system from generating unreasonable and drastic jumps in the disease, the model introduces a state regularization constraint term based on medical rationality during training and running. This regularization term uses a pre-constructed disease evolution range table to control the speed and amplitude of state changes. For example, if a single injection should not cause a body temperature to rise by 2°C within 10 seconds, the system will add this change term to the penalty term, thereby limiting unreasonable outputs. Finally, the model output S t+1 as the new patient state will be used in subsequent steps for visual rendering, behavior evaluation, and path scheduling, etc. Since the changes in S t+1 and S t are interpretable (through the causal labels of G(·)), the system can extract the key causal points of state changes for visual highlighting to assist nurses in establishing operation feedback cognition. For example, in the AR interface, "fast bolus infusion leading to blood pressure drop" will be displayed in the corresponding dimension with red flashing, allowing nurses to actively learn from physiological feedback.

[0102] Further, the calculation process of the state feedback mask of the i-th state dimension includes:

[0103]

[0104] σ(·) is the threshold activation function (step function);

[0105] and represent S t+1 and St the i-th state dimension in Δ

[0106] α i is the clinical risk weighting coefficient of the i-th state dimension, for example, set the clinical risk weighting coefficient as 1.5 for "respiratory rate" and set the clinical risk weighting coefficient as 0.6 for "reddish face";

[0107] τ1 i is the set dimension sensitivity threshold, (such as high sensitivity dimension setting τ1 i = 0.05);

[0108] n represents the dimension of the state vector of the patient.

[0109] Output indicates which variables in the current state have undergone significant changes that need the attention of the nurse.

[0110] The present application first calculates the difference between S t and S t+1 to determine whether a t significant change in the patient's state is triggered. Since small changes in some variables have clinical significance in the nursing scene (for example, a 3mmHg drop in blood pressure is a dangerous signal in a patient with blood loss), the present application designs different significance thresholds τ1 i for different variables, and based on these variable importance weights, generates a state feedback mask. i

[0111] Map Δ * to the AR display layer to construct a visual feedback object bound to the virtual patient. For example:

[0112] Δ * "heart rate" is marked as 1 → the patient's chest rhythmic vibration is enhanced;

[0113] "consciousness is blurred" is marked as 1 → facial expression is switched to a dull state, and voice is slow;

[0114] "skin is cold and wet" is marked as 1 → skin map is added with a flashing sweat special effect.

[0115] All feedback is based on the state dimension and visual mapping table, and the state controller drives the AR layer to render and update in real time, ensuring that the nurse can perceive the dynamic changes of the patient in the spatial field of view immediately after performing the behavior, and strengthening the immediate cognition of "behavior-consequence".

[0116] Further, based on the state feedback mask, the rationality score r t of the current behavior of the nurse is calculated, including:

[0117] The calculation function of the rationality score is as follows:​

[0118]

[0119] wherein:

[0120] r t ∈[0,1],r t is the rationality score of the current action, which is used for path optimization later;

[0121] γ i (a t ) represents the negative impact strength of a t on the i-th state latitude, from the action-state influence matrix (based on knowledge graph learning);

[0122] μ is the system adjustment term coefficient, for example, 0.1;

[0123] Ω(a t ,S t+1 ) is a consistency regularization term for action-state results, which is used to encourage those operations that are consistent with the positive expectations of the state.

[0124] For example, if a t is "language pacification", the system expects the "anxiety score" in S t+1 to decrease, and if it actually decreases, Ω provides a positive reward; otherwise, it deducts points.

[0125] All behavior scores will be fed back to the nurse through voice + visual prompts, and a natural language causal explanation will be generated by the large model. For example: "Due to the change of body position during the fluctuation of the patient's blood pressure, the blood pressure dropped rapidly, it is recommended to stabilize the blood pressure first and then operate." The language generation module fills in the template automatically based on the structure field in Δ * and a t , combines the high-weight dimensions in the state change with the semantic labels to ensure that the content is controllable, contextually consistent, and provides an "operation suggestion" field for training review.

[0126] Step three aims to convert the new state vector S t+1 of the patient obtained by reasoning in step two into a multi-modal interactive feedback that can be perceived, evaluated, and learned, to help the nurse understand the specific impact of her behavior a t on the patient's state. This process is not only the presentation of the model's reasoning results, but also the core bridge for the nurse to form causal cognition and establish the connection between operation and consequences.

[0127] Further, step four includes:

[0128] The candidate task library represents M is the total number of candidate tasks;

[0129] The candidate task τi The calculation formula of the priority score of is as follows:

[0130]

[0131] ψ(τ i ) is the adaptation priority score of the candidate task τ i .

[0132] represents the state feedback mask of the jth state dimension;

[0133] β j represents the importance weight of the jth state dimension, which is determined by the medical atlas; according to the strength of the causal relationship between different state variables and nursing behaviors in the medical knowledge atlas, the importance of each state dimension in decision training is calculated. The specific method is to count the number of causal edges directly connected to each state dimension in the atlas and the relationship type (such as high risk, key trigger, etc.), and combine the weight rules to score, and finally normalize the score as the weight of the dimension, which is commonly used in the range of 0.1 to 1.0.

[0134] If the number of times the jth state dimension has appeared in the error frequent state indicator record ε is greater than the set threshold (for example, 10% of the total number of records in ε), is 1, otherwise 0;

[0135] λ is the incentive coefficient for matching the error dimension (such as set to 0.3);

[0136] ρ is the coefficient for punishing repeated training; ρ is generally set between 0.1 and 0.5, and is determined according to the frequency of similar tasks in the historical task records of nurses:

[0137] If the system detects that a certain type of task is repeated frequently, ρ can be increased (such as set to 0.4-0.5) to enhance the punishment effect; if the training task coverage is still insufficient, ρ can be reduced (such as set to 0.1-0.2) to encourage basic training consolidation. The final value can be determined by adjusting the parameters in a small range to verify the diversity and coverage of the training path.

[0138] Redundancy(·) represents the similarity of the candidate task τ i to the historical training task .

[0139] The candidate task with the highest priority score is selected from all candidate tasks as the task τ1 t+1 of the next round of training.

[0140] ​Step four ensures that the system not only reacts to the current performance of the nurse (such as error correction), but also uses historical error frequency and training path coverage for global planning, making the training directional rather than short-term error correction. At the same time, by explicitly introducing a state feedback mask into task selection, a close loop between operation results and task generation is realized, so that each training is based on real feedback results.

[0141] Further, step five includes:

[0142] τ1 t+1 Contains:

[0143] Initial state vector Configure the patient's basic state at the beginning of the new task (e.g., heart rate = 120, blood pressure = 85 / 60);

[0144] Operation target list: contains predefined training targets (e.g., "perform oxygen inhalation + soothing language + body position adjustment");

[0145] Behavior type label: used for task definition (e.g., "assessment type" or "treatment type");

[0146] Context parameter field: used for task interaction loading (e.g., "emergency", "elderly patient", "language barrier", etc.);

[0147] According to Start the virtual patient state configuration and activate the corresponding vital sign simulation module (such as ECG, respiratory rate, expression, voice response);

[0148] Synchronize the operation target list and context parameter field in τ1 t+1 to the nurse interaction layer, including:

[0149] AR interface target prompt (such as superimposed label "please handle patient breathing difficulty");

[0150] Environment simulation loading (such as ICU scene background, device sound effect, etc.);

[0151] Interactive entity activation (such as virtual oxygen inhalation device, bed button).

[0152] After the nurse performs the next round of training τ1 t+1 , record the video image frame sequence v t+1 , the voice signal s t+1 and the interaction event e t+1 of this operation;

[0153] Concatenate v t+1 , s t+1 and e t+1 into a unified dimensional feature representation xt+1 ;

[0154] generate a new round of structured semantic behavior vector a t+1 :

[0155] a t+1 = NurseActionSC(x t+1 )

[0156] a t+1 is the direct input of state reasoning in the subsequent step.

[0157] The output of this step is a new round of structured semantic behavior vector a t+1 , which will be input into f t+1 along with the current state S θ in the next round of state reasoning to build the next stage response of the patient. Since the task τ t+1 is designed based on the previous round r t and Δ * , a t+1 is a direct response behavior to the performance of the last round, which is the starting point of the behavior of the system to form a complete closed-loop learning mechanism.

[0158] On the other hand, the present application provides a medical and health care decision-making training system based on a large model and augmented reality, comprising a generation module, a first acquisition module, a first calculation module, a second calculation module and a second acquisition module;

[0159] The generation module is used to collect the video image frame sequence v t collected by the AR headset, the voice signal s t issued by the nurse and the recorded interaction event e t after the nurse enters the training scene, generate a structured semantic behavior vector a t based on v t , s t and e t ;

[0160] The first acquisition module is used to acquire the state vector S t of the current patient and the individual parameter vector c of the patient, input S t , c and a t into a state generation master model to obtain a new state vector S t+1 of the patient;

[0161] The first calculation module is used to generate a state feedback mask of each state dimension based on S t and S t+1 , and calculate a rationality score r t of the current behavior of the nurse based on the state feedback mask;

[0162] The second calculation module is configured to calculate a priority score of the candidate task based on the state feedback mask, and determine a task τ1 for the next round of training based on the priority score t+1 ;

[0163] The second acquisition module is configured to acquire a structured semantic behavior vector a of the next round of training based on τ1 t+1 t+1 .

[0164] In view of the problems of the existing nursing training system, such as patient state static, lack of real-time feedback, errors cannot cause consequences, and rigid path training, the application proposes a nursing decision training method and system combining large model reasoning and augmented reality feedback mechanism. First, by introducing a large language model with medical knowledge understanding ability, the application can dynamically infer the state change of the virtual patient according to the operation behavior of the nurse in the training scene, overcoming the problems of fixed illness change and lack of response in the traditional system. Secondly, by combining with AR technology, the system can truly present the patient's physical sign evolution process on the visual level, such as skin color, consciousness state, vital signs, etc., realize the interactive closed loop between the nurse's behavior and the patient's feedback, and improve the immersion and causal cognition ability of the training. Further, the application also introduces a simulation mechanism and emergency feedback design for nursing errors, so that the nurse can observe the specific illness changes caused by the error after the operation error, and perform multiple rounds of remedial operation training, solving the one-sided problem of "error is failure and cannot be corrected" in the existing training. Finally, the system can dynamically adjust the training path combined with the training history and current performance of the nurse, realize personalized, multi-round and adaptive training mode, and enhance the decision-making ability and operation agility of the nurse in complex continuous nursing scene. The above innovations have broken through the existing nursing training technology in system, interaction and intelligence, and have significant practical application value and engineering implementation possibility.

[0165] The above is only a specific embodiment of the application, but the protection scope of the application is not limited thereto. Any skilled person in the art can easily think of changes or replacements within the technical range disclosed by the application, which should be covered within the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims.​

Claims

1. A method for medical care decision training based on a large model and augmented reality, characterized in that, include: Step one, after the nurse enters the training scene, collect the video image frame sequence collected by the AR head-mounted display , the voice signal sent by the nurse , and the recorded interaction event , generate a structured semantic behavior vector based on , , and ;​ Step two, obtain the state vector of the current patient , the individual parameter vector of the patient , input , and to the state generation master model to obtain the new state vector of the patient ; Step three, generating a state feedback mask for each state dimension, calculating a reasonableness score for the current behavior of the nurse based on the state feedback mask and generating a state feedback mask for each state dimension, calculating a reasonableness score for the current behavior of the nurse based on the state feedback mask ; Step four, calculating a priority score of the candidate task based on the state feedback mask, determining the task for the next round of training based on the priority score ; Step 5, based on Obtain the structured semantic behavior vector for the next round of training ; The calculation formula is: , Let be the state vector of the new patient, representing the virtual patient's response after the nurse's action; Generate a master model for the state and perform multidimensional variable prediction based on the Transformer structure; This is a causal regulation function used to rearrange and correct states based on the difference between the medical causal graph and the current state. To control the coefficient of the causal enhancement term.

2. The medical and health care decision-making training method based on large models and augmented reality according to claim 1, characterized in that, Step one includes: Will , and Concatenated into a feature representation of a unified dimension ; The structured semantic coding module NurseActionSC is used to process the feature representation sequence of the three most recent time steps with a unified dimension. Perform context aggregation and output structured semantic behavior vectors. .

3. The medical and health care decision-making training method based on large models and augmented reality according to claim 2, characterized in that, Including basic vital signs, functional status, and environment-dependent variables, individual parameter vectors This includes age, gender, allergy history, and chronic disease labels.

4. The medical and health care decision-making training method based on large models and augmented reality according to claim 1, characterized in that, The calculation formula is: , The causal regulation weight matrix is ​​composed of a state vector and a structured semantic behavior vector. This represents concatenating the current state vector with the structured semantic behavior vector; This is an activation function used to restrict the output to only express an upward adjustment trend, preventing unreasonable reverse adjustment; for Dimensions.

5. The medical and health care decision-making training method based on large models and augmented reality according to claim 4, characterized in that, State feedback mask for the i-th state dimension The calculation process includes: ； Use a threshold activation function; They represent and The first in One state dimension; For the first Clinical risk weighting coefficients for each state dimension; The set dimension-sensitive threshold; n represents the dimension of the patient's state vector.

6. The medical and health care decision-making training method based on large models and augmented reality according to claim 5, characterized in that, Step four includes: Candidate task library representation M represents the total number of candidate tasks; Candidate Tasks The formula for calculating the priority score is: For candidate tasks Prioritize scores based on compatibility. This represents the state feedback mask for the j-th state dimension; The importance weight of the j-th state dimension is determined by the medical atlas; If the j-th state dimension was previously recorded in the error-frequent state index... The number of times it appears is greater than the set threshold. It is 1 if it is true, otherwise it is 0. The incentive coefficient for matching the incorrect dimension; This is the coefficient used to penalize repeated training; Indicate candidate tasks With historical training mission Similarity; Select the candidate task with the highest priority score from all candidate tasks as the task for the next round of training. .

7. The medical and health care decision-making training method based on large models and augmented reality according to claim 6, characterized in that, Step five includes: Include: Initial state vector , Configure the patient's baseline condition when starting a new task; Operation target list: contains predefined training targets; Behavior type tags: used for task limitation; Context parameter field: used for task interaction loading; according to Start the virtual patient status configuration and activate the corresponding vital signs simulation module; Will The operation target list and context parameter fields are synchronized to the nurse interaction layer. Nurses are undergoing the next round of training. Then, record the video image frame sequence of this operation. The voice signals issued by the nurse and interactive events ; Will , and Concatenated into a unified dimension feature representation for the next round of training ; Generate a new round of structured semantic behavior vectors : It serves as the direct input for state reasoning in subsequent steps.

8. A medical and health care decision-making training system based on large models and augmented reality, characterized in that: It includes a generation module, a first acquisition module, a first calculation module, a second calculation module, and a second acquisition module; The generation module is used to collect video image frame sequences captured by the AR headset after the nurse enters the training scene. The voice signals issued by the nurse and recorded interaction events ,based on , and Generate structured semantic behavior vectors ; The first acquisition module is used to acquire the current patient's state vector. Patient's individual parameter vector ,Will , and Input to the state generation master model to obtain the patient's new state vector ; The first calculation module is used for... and Generate state feedback masks for each state dimension, and calculate the reasonableness score of the nurse's current behavior based on the state feedback masks. ; The second calculation module is used to calculate the priority score of candidate tasks based on the state feedback mask, and to determine the tasks to be used for the next round of training based on the priority score. ; The second acquisition module is used for... Obtain the structured semantic behavior vector for the next round of training ; The calculation formula is: , Let be the state vector of the new patient, representing the virtual patient's response after the nurse's action; Generate a master model for the state and perform multidimensional variable prediction based on the Transformer structure; This is a causal regulation function used to rearrange and correct states based on the difference between the medical causal graph and the current state. To control the coefficient of the causal enhancement term.

Images