Nurse team cooperation training system combined with virtual reality technology

Abstract

The invention belongs to the technical field of teaching based on the virtual reality technology, and discloses a nurse team cooperation training system combined with the virtual reality technology. A medical process cooperative training decision chain is constructed based on medical process cooperative training full-process parameters, medical process key actions are judged, multi-modal fusion is performed on the medical process key actions, and a micro-action recognition result is converted into a judgeable data structure based on a micro-action action template. And correcting the medical process cooperative training decision chain based on the recognition result of the micro-action. Fine-grained splitting of key actions of a medical process is introduced, action templates of the micro-actions are established for multiple groups of micro-actions, and recognition results of the micro-actions are converted into a data structure capable of being judged on the basis of the action templates of the micro-actions. The method has the advantages of comprehensiveness and accuracy, and the accuracy of key operation identification is improved.

Description

Technical Field

[0001] This invention relates to the field of teaching technology based on virtual reality technology, and more specifically, to a nurse team collaboration training system that incorporates virtual reality technology. Background Technology

[0002] With the continuous development of virtual reality technology, collaborative training has been upgraded from "experience-based" to "repeatable, quantifiable, immersive, and risk-controllable." Virtual reality technology allows for unlimited training sessions, the inclusion of different unexpected events, and the adjustment of stress levels, which can better improve the team's response speed and communication efficiency. However, the technology (especially the accuracy of motion capture) still limits its role in fine motor skills training.

[0003] Combining nurse teamwork training with virtual reality technology is beneficial, but current techniques often prioritize individual skills training, neglecting the importance of teamwork. In teamwork, a complete decision-making process consists of multiple discrete actions, all of which must be completed before moving to the next stage. Missing any action can block the decision chain. Because motion tracking heavily relies on vision or inertial navigation and is highly sensitive to occlusion and rapid movement, virtual reality technology struggles to accurately recognize minute movements. This causes the system to repeatedly validate discrete actions within the same decision-making process. If a rescue scenario requires multiple medical personnel... The lack of coordination in this process exacerbates the blockage in the decision-making chain. For example, an emergency patient experiences respiratory distress and requires rapid and effective ventilation. Nurse A uses the head-tilt / chin-lift maneuver to maintain an open airway. At this point, another nurse, B, needs to prepare oxygen and breathing equipment. If the patient has nasal and oral secretions, nurse C needs to assist nurse A in establishing the position and clearing the secretions. The actions of all three must simultaneously satisfy the following conditions for the system to determine that ventilation has been established: whether the airway has been successfully opened by A, whether B's breathing bag is properly connected and the oxygen source is on, and whether C has promptly cleared oral secretions. However, A's actions are too subtle, and virtual reality technology cannot recognize A's actions, resulting in A's actions being missing, and thus the system determines that ventilation has not been established. This misjudgment leads to the system's inability to accurately determine the completion status of the team's tasks, thereby blocking subsequent treatment decision chains, causing the evolution of the patient's condition to become disconnected from the team's actions, and making it impossible to build a realistic and reliable team collaboration training process.

[0004] In view of this, the present invention proposes a nurse team collaboration training system that incorporates virtual reality technology to solve the above problems. Summary of the Invention

[0005] To overcome the aforementioned deficiencies of the prior art and to achieve the above objectives, the present invention provides the following technical solution: a nurse team collaboration training system incorporating virtual reality technology, comprising: The decision chain construction module is used to obtain the full-process parameters of medical process cooperation training, and to construct the medical process cooperation training decision chain based on the full-process parameters of medical process cooperation training. The full-process parameters of medical process cooperation training include key decision nodes, triggering conditions of key decision nodes, medical verification actions and time constraints. The key action identification module is used to extract all actions of one of the decision processes in the medical process collaborative training decision chain, list the causal relationships of all actions, identify key actions of the medical process, and perform multimodal fusion on key actions of the medical process. The identification and optimization module is used to break down the key actions of the medical process into fine-grained parts, obtain multiple sets of micro-actions of the key actions of the medical process, extract observable features of the micro-actions, establish action templates for the micro-actions, and transform the identification results of the micro-actions into a judgmentable data structure based on the action templates. The judgmentable data structure includes the completion status of the micro-action, the accuracy of the micro-action, and the temporal information of the micro-action. The decision chain correction module is used to correct the decision chain of medical process collaboration training based on the recognition results of micro-actions, and to continue team collaboration training based on the corrected decision chain of medical process collaboration training.

[0006] Preferably, a decision chain for collaborative medical process training is constructed based on parameters of the entire collaborative medical process training process, including: Collect key decision-making nodes, triggering conditions, medical validation actions, and time constraints in the nurse team collaboration training process. Extract key decision dependencies from key decision nodes, construct a mandatory sequence chain based on time constraints and key decision dependencies, construct a conditional branch chain based on the triggering conditions of key decision nodes, and construct a branch jump chain based on the success or failure of medical verification actions. Integrate mandatory sequence chains, conditional branching chains, and branch jump chains to construct a collaborative training decision chain for medical processes.

[0007] Preferably, multimodal fusion is performed on key actions in the medical process, including: Time alignment is performed on all modalities, multimodal data of key actions in the medical process are collected, intermediate features of the multimodal data are extracted, the intermediate features of all multimodalities are concatenated into a feature vector, each modality is independently model-encoded to obtain the semantic vector of the modality, and the feature vector and semantic vector are adaptively weighted and fused to obtain the completion degree of key actions in the medical process. The intermediate features include position features, motion features, and posture features.

[0008] Preferably, the key actions in the medical process are broken down into fine-grained steps, including: Identify the necessary stages of key actions in the medical process, obtain the lower bound of motion acquisition in virtual reality technology, and break down the necessary stages of key actions in the medical process into multiple micro-actions with smaller granularity based on the lower bound of motion acquisition. Then, summarize the micro-actions of each stage into a micro-action set.

[0009] Preferably, the observable features of micro-movements are extracted, and a micro-movement action template is established, including: Determine the influence area of ​​micro-actions, extract observable features of micro-actions from the influence area, perform time-series slicing on the observable features, and construct observable feature vectors of micro-actions. The observable feature vectors of the micro-movements include attitude features, dynamic features, and mechanical features; A feature space for micro-actions is constructed based on the observable feature vectors of micro-actions, and static action templates and dynamic work templates are constructed in the feature space of micro-actions. The static work template is suitable for posture-related micro-movements, while the dynamic work template is suitable for process-related micro-movements. The work template includes the template type, the judgment conditions for action completion, and the default time length.

[0010] Preferably, the micro-action recognition results are transformed into a judgmentable data structure based on the micro-action action template, including: Each micro-action is matched with a suitable working template for micro-actions, and the accuracy of the micro-actions is calculated based on the matching degree. The expected time window for micro-actions is set based on the default time length. The difference between the action completion time and the expected time window for micro-actions is analyzed to obtain the timing information of micro-actions. The completion status of micro-actions is determined based on their accuracy and timing information.

[0011] Preferably, the decision chain for collaborative training in medical processes is corrected based on the recognition results of micro-actions, including: The thresholds, condition terms, and sequential dependencies of the triggering conditions for key decision nodes are dynamically adjusted based on the accuracy of micro-actions and the timing information of micro-actions. A multi-verification mechanism is established for key decision nodes based on the completion status of actions. If multiple micro-actions simultaneously meet the requirement that the accuracy rate of micro-actions is higher than the threshold, the key decision is completed. If the verification fails, the medical verification action is judged to be repeated by accumulating the completion status of micro-actions and the timing information of micro-actions. The time window for entering the next decision node is delayed based on the accuracy of micro-actions, and the time window for actions is automatically extended according to the completion status of the actions.

[0012] Preferably, team collaboration training continues based on the medical process collaborative training decision chain after micro-action correction, including: After correcting the medical process collaboration training decision chain based on the micro-motion recognition results, preset thresholds are set for micro-motion recognition accuracy, key decision node trigger rate, medical verification action pass rate, and time constraint satisfaction. The correction process is repeated iteratively until the micro-motion recognition accuracy, key decision node trigger rate, medical verification action pass rate, and time constraint satisfy the thresholds. The iteration stops, and the corrected medical process collaboration training decision chain is generated.

[0013] Preferably, the feature vector and semantic vector are adaptively weighted and fused, including: The feature vector and semantic vector are initially concatenated, and the weight score of each modality is calculated. An adaptive weight fusion model is established using an attention mechanism. The concatenated feature vector and semantic vector are weighted and summed according to the modality weights to obtain the fusion vector. The fusion vector is then matched with the standard action template of the key actions in the medical process to obtain the completion degree of the key actions in the medical process.

[0014] Preferably, the thresholds, condition terms, and sequence dependencies of the triggering conditions for key decision nodes are dynamically adjusted based on the accuracy of micro-actions and the timing information of micro-actions, including: Obtain the micro-action accuracy threshold. If the micro-action accuracy is higher than the micro-action accuracy threshold, lower the threshold of the triggering condition for the key decision node. If the micro-motion accuracy rate is higher than the micro-motion accuracy rate threshold, then the necessary and sufficient condition in the triggering condition will be reduced to a necessary but not necessary condition. Calculate the deviation between the completion time of the micro-action and the standard time. If the deviation is greater than the time constraint, rearrange the dependency order.

[0015] The technical effects and advantages of the nurse team collaboration training system and method combining virtual reality technology of this invention are as follows: This invention automates the evaluation of medical process training by first constructing a collaborative training decision chain for medical processes. Key actions within this chain are broken down and analyzed in detail, with micro-actions further refined. The decision chain is then optimized and corrected based on the identification results of these micro-actions. This ensures that even if key actions in the decision chain are not clearly identified, they will not significantly impact subsequent treatment procedures. Through multimodal fusion and action template matching, this invention improves the accuracy of key operation identification. The decision chain is dynamically corrected based on the identification results, making VR team training processes more closely resemble real clinical scenarios. This addresses the disconnect between disease progression and team operations in traditional training, achieving automated evaluation of medical process training. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the nurse team collaboration training system that incorporates virtual reality technology according to the present invention; Figure 2A diagram illustrating a collaborative training method for nurse teams that incorporates virtual reality technology. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] This application provides a nurse team collaboration training system that incorporates virtual reality technology. The system's implementers include, but are not limited to, devices integrated with the system such as: VR headsets, VR positioning and tracking systems, VR touch gloves, fine-grained hand motion recognition systems, and multi-person voice recognition systems.

[0019] This invention provides a nurse team collaboration training system that combines virtual reality technology. Through decision chain construction, key action recognition, recognition optimization, and decision chain correction, it optimizes and corrects the decision chain for collaborative training in medical processes by combining the recognition results of micro-actions, thereby improving the accuracy of key operation recognition and providing automated evaluation of medical process training.

[0020] Please see Figure 1 In this embodiment of the invention, the nurse team collaboration training system incorporating virtual reality technology includes: The decision chain construction module is used to obtain the full-process parameters of medical process cooperation training, and to construct the medical process cooperation training decision chain based on the full-process parameters of medical process cooperation training. The full-process parameters of medical process cooperation training include key decision nodes, triggering conditions of key decision nodes, medical verification actions and time constraints. The key action identification module is used to extract all actions of one of the decision processes in the medical process collaborative training decision chain, list the causal relationships of all actions, identify key actions of the medical process, and perform multimodal fusion on key actions of the medical process. The identification and optimization module is used to break down the key actions of the medical process into fine-grained parts, obtain multiple sets of micro-actions of the key actions of the medical process, extract observable features of the micro-actions, establish action templates for the micro-actions, and transform the identification results of the micro-actions into a judgmentable data structure based on the action templates. The judgmentable data structure includes the completion status of the micro-action, the accuracy of the micro-action, and the temporal information of the micro-action. The decision chain correction module is used to correct the decision chain of medical process collaboration training based on the recognition results of micro-actions, and to continue team collaboration training based on the corrected decision chain of medical process collaboration training.

[0021] The modules are connected via wired and / or wireless means to enable data transmission between them.

[0022] In this embodiment of the invention, the detailed implementation steps for constructing a collaborative training decision-making chain for medical processes include: Standardized medical processes, such as cardiopulmonary resuscitation, trauma first aid, and postoperative care, are obtained from the hospital's central control system through document parsing. The roles of the team in the medical process are clarified, and the collected processes are transformed into structured full-process parameters according to the action sequence, including key decision nodes, triggering conditions for key decision nodes, medical verification actions, and time constraints. At the same time, common voice commands in the process are extracted and standardized, such as "prepare for defibrillation", "call for support", and "1mg adrenaline IV push", as the command library for subsequent voice recognition modules.

[0023] The process is analyzed step by step, and all "actions that determine the direction of the process" are extracted using NLP or manual annotation and marked as a key decision node. Each key decision node represents a behavior or judgment point that will affect the direction of the treatment process, such as: whether to call for support, whether to establish intravenous access, whether to perform hemostasis, whether to perform defibrillation, and whether to perform airway opening.

[0024] An attribute table is established for behaviors affecting the treatment process, including the decision node name, prerequisite actions, triggering conditions, and success criteria for each action. The success criteria are used as medical verification actions. For example, if the decision node is chest compressions, the verification action would be whether the compression frequency, depth, and rhythm are within the prescribed range. A "voice trigger command" field is added to the attribute table, binding the standardized voice commands from step 1 to the corresponding key decision nodes. For example, the "call for support" node can be bound to the voice trigger commands "call for support" or "need help".

[0025] The time constraints of nodes include the maximum allowable delay time, the minimum action duration, the time window range of key nodes, and the dependency sequence between nodes. The average completion time, minimum action duration, and maximum allowable delay time of each key action in real operation are statistically analyzed through standardized medical processes and used as the time constraints of nodes.

[0026] Based on the sequential dependencies and time requirements between nodes, all nodes that must be executed in order are linked together to form a mandatory sequence chain. Then, a conditional branch chain is constructed according to the triggering conditions of each node (such as action completion, changes in patient status, team member responses, etc.), so that the process can enter different operation paths when different conditions are met. At the same time, a branch jump chain is designed according to the success or failure of medical verification actions, so that the process can automatically jump back to the remedial step or enter the alternative path when verification fails. Finally, the mandatory sequence chain, conditional branch chain, and branch jump chain are integrated into a complete medical process collaboration training decision chain according to node index and logical relationship, realizing structured modeling and executable control of the entire team collaboration process.

[0027] A complete collaborative training decision-making chain for medical processes is established, providing a directly executable operational model for the virtual reality training system. This enables teams to operate according to standardized and scientific decision-making logic during training, improving the relevance and effectiveness of the training. Through structured full-process parameter and decision-making chain modeling, a quantitative description of the treatment process is achieved, solving the problems of previous training relying on human experience and verbal guidance, and lacking unified standards. In the virtual reality training system, the voice recognition module analyzes the trainees' voice commands in real time. When a command matches a "voice trigger command" at a node in the decision-making chain, the corresponding decision node is triggered, driving the process evolution and making team collaboration more closely resemble real clinical scenarios.

[0028] In this embodiment of the invention, the detailed implementation steps for extracting all actions of one decision-making process in the collaborative training decision chain of the medical process, listing the causal relationships of all actions, and determining the key actions of the medical process include: A complete decision-making process is selected from the collaborative training decision-making chain of the medical workflow. By analyzing the training process documents, all action information involved in the decision-making process is extracted, including the execution order, target, and purpose of each action, forming an action list for the decision-making process. Causal relationship analysis is performed on the actions in the list. By observing the dependencies between actions, the triggering conditions of actions, and the impact of actions on subsequent key nodes, the sequential dependencies and causal logic of each action are clarified. Based on the sequential dependencies and causal logic, an action causal relationship diagram is constructed. Based on the triggering conditions of key nodes, key actions in the medical workflow are identified—those whose absence or incorrect execution would block the decision or significantly affect the workflow's effectiveness. The extracted key actions and their causal relationships are then organized into structured data.

[0029] By extracting all actions, their execution sequence, operational objects, and action purposes from the decision-making process, and conducting causal relationship analysis to form an action causal relationship diagram, a visual and structured description of the logical relationships and dependent sequences between actions in the medical process is achieved. This solves the problem of difficulty in identifying the dependencies and causal relationships between actions in traditional training, making the role and impact path of each action clear and traceable.

[0030] In this embodiment of the invention, the detailed implementation steps for multimodal fusion of key actions in the medical process include: The data streams from all modalities are timestamped and aligned. After establishing a unified timeline, multimodal data under corresponding time slices are collected synchronously. The multimodal data includes visual data, inertial sensor data, spatial positioning data, and speech data. Intermediate features containing fine-grained changes in motion are extracted from each modality, including positional features such as the 3D coordinates of joints, motion features such as velocity and acceleration, and posture features such as joint angles and posture vectors. At the same time, acoustic features such as phonemes, speech rate, and pitch are extracted from the speech data. The intermediate features from all modalities are then concatenated according to time slices to generate a unified feature vector. Simultaneously, a separate visual encoding network is constructed for each modality to extract high-level semantic vectors for the modality, representing the semantic understanding of the current key action. A dedicated speech semantic encoding network is also constructed for the speech modality to transform acoustic features into semantic vectors containing instruction content and emotional state. The feature vector and semantic vector are then initially concatenated. An adaptive weight fusion model is constructed by introducing an attention mechanism. The weight scores of each modality are calculated using the concatenated vectors and normalized to obtain the dynamic weights of different modalities in the current key action evaluation. The concatenated vectors of each modality are weighted and summed according to their corresponding weights to obtain a fusion vector that can simultaneously represent action details and semantic information.

[0031] The process of extracting intermediate features containing fine-grained changes in motion from each modality involves the following steps: jointly processing the video modality and the depth modality using a human pose estimation algorithm to obtain fine-grained position features, fine-grained motion features, and fine-grained pose features containing fine-grained changes in motion; specifically, firstly, the data from ordinary color cameras and depth cameras are temporally aligned and spatially calibrated, the two-dimensional skeleton points are fused with the depth image, and the coordinates of key joint points of the human body are extracted using a human pose estimation algorithm to obtain a high-precision skeleton point sequence combining two-dimensional and three-dimensional features, thus forming fine-grained position features.

[0032] The coordinates of joints in continuous time frames are subjected to time series differentiation and smoothing. The rate of change of joint velocity, acceleration, angular velocity, and angular acceleration within a short time window is calculated. Combined with the depth change curve, the dynamic change information of the action at the microscale is obtained, thereby extracting fine-grained motion features.

[0033] Based on the topological relationships between skeleton points, joint vectors, joint angles, limb angles, and torsional angles are calculated. Combined with the angle change trends between consecutive frames, indicators such as posture stability, posture offset, and posture rotation amplitude are extracted to form fine-grained posture features. For the speech modality, Automatic Speech Recognition (ASR) technology is used to convert the speech stream into text, and keywords (such as medical instructions) are extracted from the text as intermediate features. Simultaneously, acoustic features, such as Mel-frequency cepstral coefficients (MFCC), are extracted to capture non-linguistic information such as intonation and speech rate.

[0034] The concatenated vectors of each modality are weighted and summed according to their corresponding weights. The specific steps are as follows: The intermediate features of all modalities are concatenated according to their feature dimensions within each time slice to form a unified feature vector for that time slice. For each modality, a separate encoding network is constructed to extract high-level semantic vectors. The unified feature vector is initially concatenated with the semantic vector of each modality to obtain the concatenated vector. An adaptive weight fusion model is established using an attention mechanism to concatenate vectors. Input the adaptive weight fusion model and calculate the weight score of each modality in the current key action evaluation. The concatenated vectors of each modality are dynamically weighted and fused, with the specific formula as follows: In the formula, For the final fusion vector, This represents the number of modes.

[0035] The fused vector is matched with the standard action templates of key actions in the medical process to obtain the completion degree of key actions in the medical process. The specific matching process is as follows: The fusion vector is obtained by weighting and summing the concatenated vectors of each modality according to their corresponding weights. Extract template vectors from standard action templates ; The dot product of the fusion vector and the template vector is calculated using the following formula: ; Calculate the norms of the fused vector and the template vector separately, using the following formula: , In the formula, For the norm of the fused vector, Let the norm of the template vector be . For which vector is being calculated, The total number of vectors.

[0036] The formula for calculating cosine similarity is as follows: The cosine similarity is mapped to the interval between 0 and 1, and then the cosine similarity is used as the completion degree of key actions in the medical process.

[0037] By aligning timestamps and unifying the timeline of visual, inertial sensor, and spatial positioning data, synchronous collection of multimodal data within the same time slice was achieved. Furthermore, intermediate features of fine-grained motion changes, including position, motion, and posture features, were extracted, enabling motion representations to encompass both microscopic motion changes and high-level semantic information, thus achieving refined fusion of multimodal features. This solved the problem of asynchronous acquisition frequencies and times for different modalities, allowing motion features to be fused and analyzed within a unified time slice.

[0038] In this embodiment of the invention, the detailed implementation steps for breaking down key actions in the medical process into fine-grained steps include: Each critical action in the medical process is analyzed by observing historical training videos, expert experience, and standard operating procedures to identify the core stages or operational steps necessary to complete the critical action, as well as the key voice commands that occur simultaneously with these stages, thus obtaining the necessary stage information for each critical action. Combining the motion acquisition capabilities of the virtual reality system, a lower bound for motion acquisition is determined—the smallest unit of motion that the system can stably acquire and recognize—through sensor resolution, sampling frequency, and human joint tracking accuracy. Based on this lower bound, each necessary stage is further broken down into multiple micro-actions of smaller granularity, ensuring that each micro-action can be accurately acquired by the virtual reality device and fully express the intent of the action at that stage. Simultaneously, key voice commands closely associated with that stage are also considered as independent micro-action units. The micro-actions extracted from the same stage are then aggregated to form a set of micro-actions for that stage.

[0039] By analyzing the core stages and operational steps of each key action and combining this with the motion capture capabilities of the virtual reality system, each stage is broken down into a set of identifiable micro-movements. This achieves a fine-grained, structured representation of key actions, making the action description both complete and quantifiable. In traditional virtual reality training, key actions are usually identified as a whole, ignoring subtle internal changes, resulting in low accuracy. Fine-grained decomposition breaks down key actions into identifiable micro-movements, solving the problems of complex actions and the difficulty in capturing micro-movements. By incorporating voice commands into the micro-movement set, synchronous structuring of "listening" and "doing" is achieved, enabling training evaluation to more comprehensively reflect the team's communication and operational synergy.

[0040] In this embodiment of the invention, the detailed implementation steps for extracting observable features of micro-movements and establishing micro-movement action templates include: Virtual reality technology is used to capture body parts and manipulated object areas affected by micro-movements. Observable features of these micro-movements are extracted from the affected areas, including postural features (such as joint angles and relative limb positions), dynamic features (such as velocity, acceleration, and angular velocity), and mechanical features (such as force / pressure values ​​of the hand or manipulated tool). Time-series slices are created based on the action execution time, and features within each slice are aggregated to form an observable feature vector for the micro-movement. A feature space for the micro-movements is constructed using all these vectors, and action templates are designed based on this space. Static action templates describe the completion state of postural micro-movements, while dynamic action templates describe the execution process of procedural micro-movements. Each template includes a template type, action completion criteria, and a default time length to determine whether a micro-movement is complete during training or evaluation and to provide a standardized reference for action recognition and completion calculation. These static and dynamic templates are compiled into a micro-movement template library, providing foundational data and judgment criteria for subsequent micro-movement recognition, multimodal fusion, and training decision chain optimization.

[0041] The specific steps for constructing a feature space for micro-motions based on their observable feature vectors are as follows: First, summarize the features within each time slice to generate observable feature vectors for the micro-motions. By collecting all observable feature vectors of micro-movements, a micro-movement feature space is constructed. In the feature space, each vector represents the characteristic state of a micro-action in a time slice or state. The feature space covers the global changes and micro-details of action execution. In the feature space, posture-type micro-action vectors are clustered, a default time length is set, and a static template is formed. Time normalization is used to perform time series modeling on the feature sequence of process-type micro-actions. The time sequence of action execution is recorded in the template, and an allowed time window is set to form a dynamic template. All static action templates and dynamic work templates are organized according to action category, stage, or micro-action number to form a micro-action template library.

[0042] By constructing a feature space from the observable feature vectors of each micro-action, and generating static action templates and dynamic working templates within this feature space, a standardized representation of micro-actions is achieved, making the action completion state and execution process quantifiable and comparable. This solves the problem of the difficulty in standardizing micro-actions.

[0043] In this embodiment of the invention, the detailed implementation steps for converting the recognition results of micro-actions into a judgmentable data structure based on micro-action action templates include: The observable feature vector of each micro-motion is matched with its corresponding micro-motion template. The cosine similarity between the feature vector and the template in terms of attitude, dynamics, and mechanics is calculated, and the micro-motion accuracy is calculated accordingly, reflecting the degree of conformity between the micro-motion and the standard motion template. Based on the preset default time length in the template, an expected time window for the micro-motion is set. The difference between the actual motion completion time and this expected time window is analyzed to obtain the timing information of the micro-motion, including whether it is ahead of schedule, delayed, or meets expectations. Combining the accuracy and timing information of the micro-motion, the completion status of the micro-motion is determined according to preset rules.

[0044] The difference analysis is performed between the actual completion time and the expected time window. The specific steps are as follows: For each micro-movement, the start time of the movement is recorded using a virtual reality system or sensors. and completion time Calculate the actual time to complete the action. Based on the default time length or desired time window preset in the micro-motion work template, the desired start time is obtained. and expected end time Forming an action time window Calculate the deviation between the actual completion time and the expected time window. , Define the micro-motion time deviation index ,like or If the timing state is determined to be completed ahead of schedule, then... If the timing state is determined to be delayed, obtain the tolerance error threshold. ,like The timing state is judged to be in line with expectations.

[0045] By calculating the deviation between the actual completion time of an action and the expected time window, precise quantification of micro-movements in the time dimension is achieved, clarifying whether the action is ahead of schedule, delayed, or as expected, thus providing quantifiable time basis for action completion assessment. This solves the problem of lacking time reference in action completion assessment.

[0046] In this embodiment of the invention, the detailed implementation steps for correcting the decision chain of collaborative training in medical processes based on the recognition results of micro-actions include: In each round of team collaboration training, the decision chain of the current medical process collaboration training is analyzed based on the completion status, accuracy, and timing information of micro-actions output by the micro-action recognition module, and the recognition status and timing information of key voice commands output by the speech recognition module. The recognition results are compared with preset micro-action recognition accuracy, key decision node trigger rate, medical verification action pass rate, and time constraints to identify deviations and deficiencies. Based on the micro-action accuracy threshold, if the micro-action accuracy of a key action is higher than the threshold, the threshold of the trigger condition for that key decision node is dynamically lowered, and the sufficient and necessary conditions in the trigger conditions are adjusted to sufficient but not necessary conditions, thus allowing for a more flexible triggering mechanism. If the recognition accuracy of a key voice command is also consistently higher than the threshold, the voice command can be used as an alternative or priority trigger condition to further improve the naturalness of the process. The deviation between the actual completion time of the micro-action and the standard time is calculated. If the deviation exceeds the time constraint, the dependency order is rearranged, and the sequential relationship between key decision nodes is adjusted to ensure that the process sequence is reasonable and the time constraints are met. By integrating the completion status of micro-actions, adjusted triggering conditions, and rearranged sequence dependencies, a revised medical process collaboration training decision chain is generated. This revision process is repeated for multiple iterations. After each training round, the accuracy of micro-action recognition, the trigger rate of key decision nodes, the pass rate of medical verification actions and time constraints, as well as the recognition accuracy and response timing of key voice commands, are re-evaluated until all indicators meet preset thresholds. The iteration then stops, ultimately yielding the optimized and revised medical process collaboration training decision chain.

[0047] By iteratively analyzing the completion status, accuracy, and timing information of micro-actions, the triggering conditions and dependency order of key decision nodes are dynamically adjusted to optimize the decision chain for collaborative training in medical processes. This ensures a reasonable process sequence, flexible triggering mechanisms, and compliance with time constraints. It solves the problem of static, fixed decision chains being unsuitable for real-world operations. By incorporating the recognition performance of voice commands into the iterative optimization loop, the decision chain not only adapts to increased operational proficiency but also to the evolution of team communication methods, achieving a human-machine collaborative process that more closely resembles real-world clinical scenarios.

[0048] like Figure 2 The diagram shown is a flowchart of team collaboration training based on the decision chain of medical process collaboration training after micro-action correction. In this embodiment of the invention, the detailed implementation steps of team collaboration training based on the decision chain of medical process collaboration training after micro-action correction include: In each round of team collaboration training, micro-motion data is collected through a virtual reality system. The micro-motion recognition module obtains the completion status, accuracy, and timing information of each micro-motion. These recognition results are compared with the triggering conditions of key decision nodes, medical verification actions, and time constraints in the current medical process collaboration training decision chain. The accuracy of micro-motion recognition, the triggering rate of key decision nodes, the pass rate of medical verification actions, and the time constraints are analyzed to determine if they meet the standards. Based on the comparison results, the decision chain is corrected, for example, by adjusting the threshold of key decision node triggering conditions, modifying condition logic, rearranging the node dependency order, or updating time constraints, to make the micro-motion recognition results more closely match the requirements of the decision chain. Then, the corrected decision chain is used in the next round of team collaboration training, and micro-motion data is collected again for recognition and evaluation. This correction and evaluation process is repeated, continuously optimizing the decision chain through multiple iterations. After each iteration, the decision chain is updated until the accuracy of micro-motion recognition, the triggering rate of key decision nodes, the pass rate of medical verification actions, and the time constraints all reach preset thresholds, at which point the iteration stops.

[0049] By comparing and analyzing the micro-action recognition results with the decision chain parameters, dynamic adjustments can be made to the triggering conditions, node dependency order, and time constraints of key decision nodes. This enables the medical process collaborative training decision chain to adapt to actual operational situations, ensuring a reasonable process sequence, accurate action triggering, and satisfactory time constraints. This solves the problem that static decision chains cannot adapt to actual operations.

[0050] This invention achieves the optimization and correction of the decision chain for collaborative training in medical processes by combining the identification results of micro-actions through decision chain construction, key action identification, identification optimization, and decision chain correction. This improves the accuracy of key operation identification and provides automated evaluation of medical process training.

[0051] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0052] It should be noted that all formulas in this manual are calculated by removing dimensions and taking their numerical values. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0053] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A nurse team collaboration training system incorporating virtual reality technology, characterized in that: include: The decision chain construction module is used to obtain the full-process parameters of medical process cooperation training, and to construct the medical process cooperation training decision chain based on the full-process parameters of medical process cooperation training. The full-process parameters of medical process cooperation training include key decision nodes, triggering conditions of key decision nodes, medical verification actions and time constraints. The key action identification module is used to extract all actions of one of the decision processes in the medical process collaborative training decision chain, list the causal relationships of all actions, identify key actions of the medical process, and perform multimodal fusion on key actions of the medical process. The identification and optimization module is used to break down the key actions of the medical process into fine-grained parts, obtain multiple sets of micro-actions of the key actions of the medical process, extract observable features of the micro-actions, establish action templates for the micro-actions, and transform the identification results of the micro-actions into a judgmentable data structure based on the action templates. The judgmentable data structure includes the completion status of the micro-action, the accuracy of the micro-action, and the temporal information of the micro-action. The decision chain correction module is used to correct the decision chain of medical process collaboration training based on the recognition results of micro-actions, and to continue team collaboration training based on the corrected decision chain of medical process collaboration training.

2. The nurse team collaboration training system combining virtual reality technology according to claim 1, characterized in that, The aforementioned decision-making chain for collaborative medical process training, constructed based on parameters across the entire medical process, includes: Collect key decision-making nodes, triggering conditions, medical validation actions, and time constraints in the nurse team collaboration training process. Extract key decision dependencies from key decision nodes, construct a mandatory sequence chain based on time constraints and key decision dependencies, construct a conditional branch chain based on the triggering conditions of key decision nodes, and construct a branch jump chain based on the success or failure of medical verification actions. Integrate mandatory sequence chains, conditional branching chains, and branch jump chains to construct a collaborative training decision chain for medical processes.

3. The nurse team collaboration training system combining virtual reality technology according to claim 1, characterized in that, The multimodal fusion of key actions in the medical process includes: Time alignment is performed on all modalities, multimodal data of key actions in the medical process are collected, intermediate features of the multimodal data are extracted, the intermediate features of all multimodalities are concatenated into a feature vector, each modality is independently model-encoded to obtain the semantic vector of the modality, and the feature vector and semantic vector are adaptively weighted and fused to obtain the completion degree of key actions in the medical process. The intermediate features include position features, motion features, and posture features.

4. The nurse team collaboration training system combining virtual reality technology according to claim 1, characterized in that, The process of breaking down key actions in the medical procedure into fine-grained steps includes: Identify the necessary stages of key actions in the medical process, obtain the lower bound of motion acquisition in virtual reality technology, and break down the necessary stages of key actions in the medical process into multiple micro-actions with smaller granularity based on the lower bound of motion acquisition. Then, summarize the micro-actions of each stage into a micro-action set.

5. The nurse team collaboration training system combining virtual reality technology according to claim 1, characterized in that, The process of extracting observable features of micro-movements and establishing micro-movement templates includes: Determine the influence area of ​​micro-actions, extract observable features of micro-actions from the influence area, perform time-series slicing on the observable features, and construct observable feature vectors of micro-actions. The observable feature vectors of the micro-movements include posture features, dynamic features, and mechanical features; A feature space for micro-actions is constructed based on the observable feature vectors of micro-actions, and static action templates and dynamic work templates are constructed in the feature space of micro-actions. The static work template is suitable for posture-related micro-movements, while the dynamic work template is suitable for process-related micro-movements. The work template includes the template type, the judgment conditions for action completion, and the default time length.

6. The nurse team collaboration training system combining virtual reality technology according to claim 1, characterized in that, The micro-action-based action template transforms the micro-action recognition results into a judgmentable data structure, including: Each micro-action is matched with a suitable working template for micro-actions, and the accuracy of the micro-actions is calculated based on the matching degree. The expected time window for micro-actions is set based on the default time length. The difference between the action completion time and the expected time window for micro-actions is analyzed to obtain the timing information of micro-actions. The completion status of micro-actions is determined based on their accuracy and timing information.

7. The nurse team collaboration training system combining virtual reality technology according to claim 1, characterized in that, The micro-motion-based recognition results are used to correct the decision chain for collaborative training in medical processes, including: The thresholds, condition terms, and sequential dependencies of the triggering conditions for key decision nodes are dynamically adjusted based on the accuracy of micro-actions and the timing information of micro-actions. A multi-verification mechanism is established for key decision nodes based on the completion status of actions. If multiple micro-actions simultaneously meet the requirement that the accuracy rate of micro-actions is higher than the threshold, the key decision is completed. If the verification fails, the medical verification action is judged to be repeated by accumulating the completion status of micro-actions and the timing information of micro-actions. The time window for entering the next decision node is delayed based on the accuracy of micro-actions, and the time window for actions is automatically extended according to the completion status of the actions.

8. The nurse team collaboration training system combining virtual reality technology according to claim 1, characterized in that, The medical process collaborative training decision chain based on micro-action correction continues team collaboration training, including: After correcting the medical process collaboration training decision chain based on the micro-motion recognition results, preset thresholds are set for micro-motion recognition accuracy, key decision node trigger rate, medical verification action pass rate, and time constraint satisfaction. The correction process is repeated iteratively until the micro-motion recognition accuracy, key decision node trigger rate, medical verification action pass rate, and time constraint satisfy the thresholds. The iteration stops, and the corrected medical process collaboration training decision chain is generated.

9. The nurse team collaboration training system combining virtual reality technology according to claim 3, characterized in that, The adaptive weight fusion of feature vectors and semantic vectors includes: The feature vector and semantic vector are initially concatenated, and the weight score of each modality is calculated. An adaptive weight fusion model is established using an attention mechanism. The concatenated feature vector and semantic vector are weighted and summed according to the modality weights to obtain the fusion vector. The fusion vector is then matched with the standard action template of the key actions in the medical process to obtain the completion degree of the key actions in the medical process.

10. The nurse team collaboration training system combining virtual reality technology according to claim 7, characterized in that, The dynamic adjustment of the thresholds, condition terms, and sequence dependencies of key decision node triggering conditions based on micro-action accuracy and micro-action timing information includes: Obtain the micro-action accuracy threshold. If the micro-action accuracy is higher than the micro-action accuracy threshold, lower the threshold of the triggering condition for the key decision node. If the micro-motion accuracy rate is higher than the micro-motion accuracy rate threshold, then the necessary and sufficient condition in the triggering condition will be reduced to a necessary but not necessary condition. Calculate the deviation between the completion time of the micro-action and the standard time. If the deviation is greater than the time constraint, rearrange the dependency order.

Images