A nursing operation virtual training intelligent guide teaching method, system, device and medium

Abstract

The application relates to a nursing operation virtual training intelligent guide method, system, device and medium. The method comprises the following steps: constructing a nursing clinical decision knowledge graph containing state nodes, operation nodes and directed edges, generating semantic embedding vectors and metadata embedding vectors; collecting multi-modal data of a patient in real time, generating an instant state vector of the patient through feature extraction and cross-modal fusion; activating a knowledge graph subgraph through similarity calculation, calculating operation node decision weights based on a graph attention network, generating visual, auditory and text multi-channel guide instructions and feeding back in real time. The method further improves the clinical adaptability of nursing operation virtual training, the guide accuracy, and the training efficiency of the clinical thinking and operation ability of students by fusing the nursing clinical decision knowledge graph, multi-modal patient state perception and intelligent decision based on the graph attention network.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, and in particular relates to an intelligent teaching method, system, device and medium for virtual training of nursing operations. Background Technology

[0002] Nursing procedures are a core component of clinical medical services, and their standardization and precision directly affect patient treatment outcomes and safety. Virtual training, as a key means to reduce clinical practice risks and improve trainees' proficiency, has been widely applied in nursing education. However, existing virtual training systems for nursing procedures largely rely on preset scripts or fixed rules to generate guidance instructions, lacking deep integration with clinical nursing decision-making logic. The training guidance models they construct often lack a structured knowledge support system, fail to effectively define the relationships between state nodes and operation nodes, and do not design key attributes such as expert rule strength, trainable edge weights, transition probabilities, and expected outcome scores for directed edges between nodes. This results in guidance instructions failing to dynamically adapt to the real-time state changes of virtual patients. Furthermore, existing virtual training systems' perception of virtual patient states is mostly limited to single-modal data acquisition and processing, failing to integrate multi-dimensional information such as vital sign values, facial expression images, audio recordings of patient complaints, and local vital sign images. They lack effective cross-modal interaction and fusion mechanisms, making it difficult to generate a comprehensive real-time state vector reflecting the patient's state, thus affecting the accuracy of guidance decisions. Furthermore, the guidance and decision-making methods during the training process are mostly based on single-channel feedback, which cannot provide trainees with real-time operational instructions from multiple dimensions such as visual, auditory, and text. This results in insufficient targeting and effectiveness of training guidance, making it difficult to help trainees quickly grasp the core logic and standardized procedures of clinical nursing operations, thus restricting the teaching effect and practical value of the virtual training system. Summary of the Invention

[0003] Based on this, it is necessary to provide a method, system, device, and medium for intelligent guidance of virtual training in nursing operations to address the aforementioned technical problems. The aim is to improve the dynamic adaptability and decision-making accuracy of virtual training guidance instructions in nursing operations, enhance the comprehensiveness and real-time nature of virtual patient status perception, and strengthen the multi-dimensional immersion and teaching effectiveness of training guidance.

[0004] Firstly, this application provides a virtual training intelligent teaching method for nursing operations, including:

[0005] Construct a nursing clinical decision knowledge graph that includes state nodes, operation nodes, and directed edges connecting state nodes and operation nodes; generate semantic embedding vectors for each state node and each operation node, and generate metadata embedding vectors for each operation node; wherein, the directed edges include first-class edges pointing from state nodes to operation nodes and second-class edges pointing from operation nodes to state nodes. The first-class edge attributes include expert rule strength, trainable edge weights, and operation precondition vectors, and the second-class edge attributes include transition probability and expected outcome score;

[0006] Multimodal data of digital twin patients in a virtual training scenario are collected in real time. The multimodal data includes vital sign values, facial expression images, audio of the patient's voice complaints, and images of local signs. Feature extraction is performed on the data in the multimodal data to obtain the corresponding single-modal feature vectors. The single-modal feature vectors are input into the multimodal fusion encoder for cross-modal interactive fusion to generate the patient's real-time state vector.

[0007] The similarity between the patient's immediate state vector and the semantic embedding vectors of each state node in the nursing clinical decision-making knowledge graph is calculated. Based on the similarity results, the corresponding state nodes and their neighboring nodes are activated to form an activation subgraph. The features of each node in the activation subgraph are initialized based on the patient's immediate state vector and the semantic embedding vectors of the corresponding state nodes in the nursing clinical decision-making knowledge graph. The activation subgraph is then input into a graph attention network with trainable parameters for multi-layer information propagation. The decision weight of each operation node is calculated, and operation nodes that meet preset conditions based on their decision weights are selected to generate multi-channel guidance instructions. These multi-channel guidance instructions are then provided to trainees in real time through visual, auditory, and text channels. The multi-channel guidance instructions are used to instruct trainees to perform corresponding nursing operations in the virtual training scenario.

[0008] In one embodiment, a nursing clinical decision-making knowledge graph is constructed, including state nodes, operation nodes, and directed edges connecting state nodes and operation nodes; a semantic embedding vector is generated for each state node and each operation node, and a metadata embedding vector is generated for each operation node, including:

[0009] Knowledge data is collected from pre-set clinical nursing guidelines, expert experience knowledge base and historical nursing cases. A biomedical entity recognition model is used to identify state description entities and operation description entities in the knowledge data to obtain the initial state node set and the initial operation node set.

[0010] By using a remote supervision method combined with a predefined rule template, the state-operation relationship and operation-state relationship between entities in the initial state node set and the initial operation node set are extracted to form the initial first type edge set and the initial second type edge set, respectively.

[0011] Semantic clustering is performed on synonymous entities in the initial set of state nodes and the initial set of operation nodes to obtain the final set of state nodes and the final set of operation nodes. Based on the entity merging results, the associations in the initial first set of edges and the initial second set of edges are adjusted synchronously to obtain the final first set of edges and the second set of edges.

[0012] Based on pre-set clinical nursing guidelines and expert experience knowledge base, the clinical effectiveness, execution priority and risk level of the state-operation relationship corresponding to each edge in the first type edge set are quantitatively scored to obtain the clinical expert score results of each first type edge; according to the clinical expert score results, the corresponding expert rule strength is determined for each first type edge in the first type edge set.

[0013] Based on the predefined precondition descriptions in the predefined rule template, the precondition descriptions are converted into vector form through a natural language processing model to generate the operation precondition vectors corresponding to each first-class edge in the first-class edge set.

[0014] Based on historical nursing case data, statistical analysis was performed to calculate the frequency of occurrence of the operation-state association relationship corresponding to each second-type edge in the second-type edge set in clinical practice, and the frequency of occurrence was used as the transition probability corresponding to each second-type edge. According to the preset clinical outcome assessment criteria, the clinical prognostic effect of the state node corresponding to each second-type edge in the second-type edge set was graded and scored, and the graded score was used as the expected outcome score corresponding to each second-type edge.

[0015] Based on the set of state nodes and the set of operation nodes, the descriptive text of each state node and each operation node is feature-encoded using a pre-trained medical BERT model to generate semantic embedding vectors for each state node and each operation node; the metadata of each operation node is feature-encoded to generate metadata embedding vectors for each operation node.

[0016] Using historical nursing case data, each historical nursing case is formatted as a state-operation transition sequence. By maximizing the log-likelihood function of expert decisions, the weights of each first-class edge in the first-class edge set are initially trained to obtain the trainable edge weights of each first-class edge.

[0017] In one embodiment, features are extracted from the multimodal data to obtain corresponding single-modal feature vectors; each single-modal feature vector is input into a multimodal fusion encoder for cross-modal interactive fusion to generate a patient's real-time state vector, including:

[0018] The vital signs values ​​are normalized to obtain normalized vital signs values. The normalized vital signs values ​​are then input into a multilayer perceptron network for feature extraction to generate vital signs feature vectors.

[0019] The facial expression image is standardized, and the standardized facial expression image is then input into a preset convolutional neural network for feature extraction to generate a facial expression feature vector.

[0020] The audio of the main complaint is segmented to obtain the segmented audio of the main complaint. The segmented audio of the main complaint is converted into text data through automatic speech recognition technology. Feature extraction is performed on the text data to generate the feature vector of the main complaint.

[0021] The local sign image is standardized, and the standardized local sign image is input into the semantic segmentation network for segmentation. Quantitative features including the proportion of red and swollen areas, the proportion of oozing areas, the probability of oozing color, and the shape factor are extracted from the segmentation results to generate a local sign feature vector.

[0022] The initial feature sequence is constructed by using the feature vectors of vital signs, facial expressions, speech complaints, and local signs as sequence elements.

[0023] Add modality type encoding and temporal position encoding to each element of the initial feature sequence to obtain the encoded feature sequence;

[0024] The encoded feature sequence is input into a multimodal fusion encoder based on the Transformer architecture. Information is exchanged and fused through multiple cross-modal self-attention layers, and the fused feature vectors of each modality are output. The mean of each feature vector is taken as the patient's real-time state vector.

[0025] In one embodiment, the similarity between the patient's immediate state vector and the semantic embedding vectors of each state node in the nursing clinical decision-making knowledge graph is calculated. Based on the similarity result, the corresponding state node and its neighboring nodes are activated to form an activation subgraph, including:

[0026] Calculate the cosine similarity between the patient's immediate state vector and the semantic embedding vector of each state node in the nursing clinical decision knowledge graph to obtain the similarity value of each state node.

[0027] Select state nodes with similarity values ​​higher than a preset similarity threshold as activation nodes; if the similarity values ​​of all state nodes are lower than the preset similarity threshold, then arrange the state nodes according to their similarity values ​​and select the first preset number of state nodes as activation nodes.

[0028] Extract the state nodes, operation nodes, and corresponding directed edges within the two-hop neighborhood of each activated node, and integrate them with the activated nodes to form an activated subgraph; wherein, the two-hop neighborhood includes the first-level neighboring nodes directly connected to the activated node, and the second-level neighboring nodes directly connected to the first-level neighboring nodes.

[0029] In one embodiment, features of each node in the activation subgraph are initialized based on the patient's immediate state vector and the semantic embedding vector of the corresponding state node in the nursing clinical decision knowledge graph. The activation subgraph is then input into a graph attention network with trainable parameters for multi-layer information propagation. The decision weight of each operation node is calculated, and operation nodes that meet preset conditions based on their decision weights are selected to generate multi-channel guidance instructions, including:

[0030] The nodes in the activation subgraph are initialized with features to obtain the initialized node features;

[0031] Extract the attribute vectors of each directed edge in the activation subgraph, input the initialized node features and the attribute vectors of each directed edge in the activation subgraph into the graph attention network, calculate the attention coefficient between each node and its corresponding neighbor nodes in the activation subgraph through a multi-head attention mechanism, and fuse the attribute vectors of the corresponding directed edges in the process of calculating the attention coefficients to obtain the final attention coefficients.

[0032] Based on the final attention coefficients, multi-layer node update processing is performed in the graph attention network, updating the feature representation of each node in the activation subgraph layer by layer, to obtain the final feature representation of each operation node in the activation subgraph and the average attention coefficients passed from each activation node in the activation subgraph to the operation node.

[0033] For each operation node in the activation subgraph, the precondition matching degree of the operation node in the activation subgraph is calculated by a preset multilayer perceptron based on the patient's immediate state vector and the corresponding semantic embedding vector. Based on the transition probability and expected outcome score of each next state node connected by the operation node in the activation subgraph through the second type of edge, and combined with the similarity between the patient's immediate state vector and the semantic embedding vector of each next state node, the neighbor expected score of the operation node in the activation subgraph is calculated.

[0034] The mean attention coefficient, the degree of matching of preconditions, and the expected score of neighbors are weighted and summed according to a preset weight ratio to obtain the decision weight of each operation node in the activation subgraph.

[0035] The operation nodes in the activation subgraph are arranged from high to low according to the decision weight. The top K operation nodes with decision weights higher than the preset weight threshold are selected to obtain the filtered operation nodes. The visual highlighting instructions, voice broadcast text and text prompt information corresponding to the filtered operation nodes are generated through the preset mapping table. The visual highlighting instructions, voice broadcast text and text prompt information are combined into multi-channel guidance instructions.

[0036] The following steps are involved in feature initialization:

[0037] For a state node in the activation subgraph, if the state node in the activation subgraph belongs to the activation node, the corresponding feature is initialized as the patient's immediate state vector; if the state node in the activation subgraph does not belong to the activation node, the corresponding feature is initialized as the semantic embedding vector of the state node itself in the activation subgraph.

[0038] For an operation node in the activation subgraph, the semantic embedding vector of the operation node in the activation subgraph is concatenated with the corresponding metadata embedding vector, and the concatenated vector is mapped to a preset dimension through a linear transformation to obtain the initial features of the operation node in the activation subgraph.

[0039] In one embodiment, the method further includes:

[0040] During the process of trainees performing nursing operations based on multi-channel guided instructions, the real-time status vectors of patients at each time step are collected according to the preset time step, arranged in chronological order, and a status sequence is generated.

[0041] Record the operation node index corresponding to the nursing operation actually performed by the trainee in the virtual training scenario at each time step, arrange them in chronological order, and generate an operation sequence;

[0042] Based on graph attention network, the state sequence is processed step by step, the decision weight of the corresponding operation node at each time step is calculated, and the decision weight sequence is generated by arranging them in chronological order. Based on the matching degree between the virtual patient's state change results after the trainee performs nursing operations based on multi-channel guidance instructions at each time step and the preset standard state change, the instant reward for the corresponding time step is calculated, and the instant reward sequence is generated by arranging them in chronological order.

[0043] After performing nursing procedures at each time step, the corresponding outcome scores are calculated based on the virtual patient's final vital signs, local signs, and pain level. These scores are then arranged in chronological order to obtain the outcome score sequence.

[0044] The state sequence, operation sequence, decision weight sequence, immediate reward sequence, and outcome score sequence are stored to construct a training trajectory database;

[0045] When the total number of trajectories accumulated in the training trajectory database reaches a preset threshold, a preset batch of four-tuple data including state, operation, reward and next state is sampled from the training trajectory database.

[0046] A Q-network with the same structure as the graph attention network is constructed. The conservative Q-learning algorithm is used to update the network parameters of the Q-network by minimizing the Bellman error, and the updated Q-network is obtained.

[0047] Using the graph attention network as the policy network, the network parameters of the policy network are updated by minimizing the KL divergence between the updated Q network and the policy network, thus obtaining the updated policy network.

[0048] The Q-values ​​of the updated Q-network output are normalized, and the trainable edge weights of the first type of edge in the nursing clinical decision knowledge graph are adjusted using the momentum update method to obtain the updated trainable edge weights.

[0049] In one embodiment, the final attention coefficient is calculated using the following formula:

[0050]

[0051]

[0052]

[0053] in, The first point of focus for the bulls Size, This represents the total number of heads of multi-head attention. For the first Node in the head For neighboring nodes The single-head raw attention coefficient; For the first Attention weight vector for each size; For the first The linear transformation matrix of the node features of each head; They are nodes ,node Initialize the feature vector; For nodes To the node The attribute vector of the directed edge; For the first Linear transformation matrix of the edge attribute features of the head; For nodes The set of neighboring nodes; The average of the multi-head attention coefficients; Use the Sigmoid activation function; This represents the final attention coefficient.

[0054] Secondly, this application also provides a smart teaching system for virtual training in nursing procedures, comprising:

[0055] The graph construction module is used to construct a nursing clinical decision knowledge graph that includes state nodes, operation nodes, and directed edges connecting state nodes and operation nodes. It generates semantic embedding vectors for each state node and each operation node, and generates metadata embedding vectors for each operation node. The directed edges include first-type edges pointing from state nodes to operation nodes and second-type edges pointing from operation nodes to state nodes. The first-type edge attributes include expert rule strength, trainable edge weights, and operation precondition vectors. The second-type edge attributes include transition probability and expected outcome score.

[0056] The state awareness module is used to collect multimodal data of digital twin patients in the virtual training scene in real time. The multimodal data includes vital sign values, facial expression images, voice complaints audio, and local vital sign images. Features are extracted from the multimodal data to obtain corresponding single-modal feature vectors. The single-modal feature vectors are input into the multimodal fusion encoder for cross-modal interactive fusion to generate the patient's real-time state vector.

[0057] The decision-guided teaching module calculates the similarity between the patient's immediate state vector and the semantic embedding vectors of each state node in the nursing clinical decision knowledge graph. Based on the similarity results, it activates the corresponding state node and its neighboring nodes, forming an activation subgraph. Using the patient's immediate state vector and the semantic embedding vectors of the corresponding state nodes in the nursing clinical decision knowledge graph as a basis, it initializes the features of each node in the activation subgraph. The activation subgraph is then input into a graph attention network with trainable parameters for multi-layer information propagation. The decision weight of each operation node is calculated, and operation nodes that meet preset conditions based on their decision weights are selected, generating multi-channel guidance instructions. These multi-channel guidance instructions provide real-time feedback to trainees through visual, auditory, and text channels. These multi-channel guidance instructions instruct trainees to perform corresponding nursing operations in a virtual training scenario.

[0058] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the first aspect.

[0059] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the first aspect.

[0060] The aforementioned intelligent guidance method, system, equipment, and medium for virtual training of nursing operations first constructs a nursing clinical decision-making knowledge graph containing state nodes, operation nodes, and multi-dimensional edge attributes, realizing the structured representation and semantic embedding of nursing knowledge and laying the knowledge foundation for intelligent guidance. Second, it collects multimodal data from digital twin patients in real time, generating real-time patient state vectors through cross-modal interaction and fusion, overcoming the limitations of traditional single-modal perception and improving the comprehensiveness and accuracy of state perception. Furthermore, based on graph attention networks, it performs multi-layer information propagation on activation subgraphs, dynamically calculates the decision weights of operation nodes, and generates multi-channel guidance instructions, achieving deep adaptation between guidance decisions and the patient's real-time state, enhancing the immersiveness and relevance of training feedback. Finally, it guides trainees' operations through multi-channel real-time feedback, improving the teaching effectiveness and practical value of the virtual training system. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 A flowchart of a virtual training intelligent teaching method for nursing operations is provided as an exemplary embodiment of the present invention;

[0063] Figure 2 A flowchart illustrating a method for constructing an activation subgraph, as provided in an exemplary embodiment of the present invention;

[0064] Figure 3 This is a schematic diagram of a smart teaching and learning system for virtual training of nursing operations, provided as an exemplary embodiment of the present invention. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0066] In one embodiment, such as Figure 1As shown, a smart guidance method for virtual training in nursing operations is provided. This embodiment focuses on applying this method to a virtual training system for nursing operations. This system can be independently deployed based on virtual training interactive terminals such as VR headsets, training hosts with touch functionality, and smart tablets, to provide localized virtual training guidance services to trainees. It is understood that this method can also be applied to cloud servers, deploying guidance decision logic and knowledge graph data to remote computing nodes via a network. Trainees access the server through terminal devices to obtain guidance services. It can also be applied to a distributed system of virtual training interactive terminals + cloud servers. The terminal completes interactive functions such as virtual scene presentation, multimodal data acquisition, trainee operation reception, and multi-channel guidance instruction feedback. The server completes computationally intensive tasks such as knowledge graph storage, multimodal data fusion calculation, and graph attention network inference. The terminal and server achieve data interaction and collaborative work through a local area network or wide area network, adapting to different scale training scenarios, such as individual self-training, centralized training in institutions, and cross-regional joint training. In this embodiment, the method includes the following steps:

[0067] S101: Construct a nursing clinical decision knowledge graph including state nodes, operation nodes, and directed edges connecting state nodes and operation nodes; generate semantic embedding vectors for each state node and each operation node, and generate metadata embedding vectors for each operation node; wherein, the directed edges include first-type edges pointing from state nodes to operation nodes and second-type edges pointing from operation nodes to state nodes. The first-type edge attributes include expert rule strength, trainable edge weights, and operation precondition vectors, and the second-type edge attributes include transition probability and expected outcome score.

[0068] Specifically, the essence of clinical nursing operations is a cyclical process of patient state - nursing operation - state change. Therefore, the patient's clinical state can be abstracted through state nodes, and standardized nursing actions can be defined through operation nodes. Directed edges connect the two, thus accurately reproducing the logical chain of clinical decision-making. Illustratively, nodes can be selected based on the core treatment pathways of clinical nursing guidelines, key operational norms of expert experience knowledge bases, and high-frequency state-operation combinations from historical nursing cases to ensure the clinical adaptability of the knowledge graph. The directed edges between nodes correspond to two core associations: state-triggered operation and operation-induced state change. The expert rule strength of the first type of edge can be used to quantify clinical recommendation priority, the trainable edge weight can reserve space for subsequent optimization, and the operation prerequisite vector can be used to judge the adaptability of the operation to the current state. The transition probability of the second type of edge reflects the possibility of state change after the operation, and the expected outcome score further quantifies the clinical value of the operation. By setting the above attributes of directed edges, the knowledge graph can break free from the limitations of traditional static rules and possess dynamic decision-making capabilities. Subsequently, a semantic embedding vector can be generated for each node, transforming the node information described in the text into a low-dimensional dense vector, enabling the machine to accurately understand the semantics of the nodes. Furthermore, by generating metadata embedding vectors for the operation nodes, it is possible to further supplement the technical parameters, execution requirements, and other detailed information of the operation, providing data support for subsequent operation adaptability judgment.

[0069] S102: Real-time acquisition of multimodal data of digital twin patients in virtual training scenarios. Multimodal data includes vital sign values, facial expression images, voice complaints audio, and local vital sign images; feature extraction of data from multimodal data to obtain corresponding single-modal feature vectors; input of each single-modal feature vector into a multimodal fusion encoder for cross-modal interactive fusion to generate a real-time patient state vector.

[0070] Specifically, a patient's clinical state cannot be fully represented by a single-dimensional data source. For example, relying solely on vital signs cannot determine the degree of pain, and facial expressions alone cannot quantify the severity of the condition. Therefore, multimodal data such as vital sign values ​​(objective physiological indicators), facial expression images (subjective states such as pain and discomfort), audio recordings of spoken complaints (symptom descriptions), and local physical signs images (local lesion features) can be selected to comprehensively understand the patient's clinical state. Subsequently, since the raw data collected, such as image pixels and audio waveforms, cannot be directly used for decision-making calculations, feature extraction can be used to transform them into single-modal feature vectors with semantic information, achieving data dimensionality reduction and abstraction. The core function of the multimodal fusion encoder is to solve the heterogeneity problem of different modal data, such as the format differences between numerical vital signs and image-based facial expression data. By inputting the feature vectors of each single modality into this encoder, cross-modal interactive fusion can eliminate information barriers between modalities, generating a unified real-time patient state vector. This vector can comprehensively reflect the current state of the digital twin patient, providing standardized input for subsequent state matching with the knowledge graph.

[0071] S103: Calculate the similarity between the patient's immediate state vector and the semantic embedding vectors of each state node in the nursing clinical decision-making knowledge graph. Based on the similarity results, activate the corresponding state node and its neighboring nodes to form an activation subgraph. Initialize the features of each node in the activation subgraph based on the patient's immediate state vector and the semantic embedding vectors of the corresponding state nodes in the nursing clinical decision-making knowledge graph. Input the activation subgraph into a graph attention network with trainable parameters for multi-layer information propagation, calculate the decision weight of each operation node, filter out operation nodes that meet preset conditions based on their decision weights, and generate multi-channel guidance instructions. Provide real-time feedback to trainees through visual, auditory, and text channels. The multi-channel guidance instructions are used to instruct trainees to perform corresponding nursing operations in the virtual training scenario.

[0072] Specifically, the sheer number of nodes and edges in a knowledge graph leads to decision-making delays due to direct computation. Therefore, a preliminary approach is to calculate the similarity between the patient's immediate state vector and the semantic embedding vectors of state nodes in the knowledge graph. This allows for the selection of activation nodes and their neighboring nodes that are highly relevant to the current state, forming an activation subgraph. This approach ensures the relevance of the decision while significantly improving computational efficiency. During node feature initialization, initializing activation nodes with the patient's immediate state vector ensures that the decision-making process closely aligns with the actual state of the current training scenario, rather than relying on the general semantics of nodes. Furthermore, fusing semantic embedding vectors and metadata embedding vectors for operational nodes allows for simultaneous consideration of the semantic and technical suitability of the operation during decision-making. Subsequently, the activation subgraph is input into a graph attention network for multi-layered information propagation. This leverages the network's attention mechanism to capture the differences in the importance of relationships between nodes. For example, when an operational node is more closely associated with an activation node, it receives a higher attention weight, thus dominating feature aggregation and achieving precise quantification of the operational node's suitability. Furthermore, decision weights can be calculated by comprehensively considering factors such as the strength of the state-operation correlation and the degree of matching between the operation and the current state. This ensures that the selected operation nodes conform to clinical logic and are suitable for the current training scenario. Multi-channel guidance instructions can be designed based on ergonomics and pedagogical principles. For example, the visual channel can use highlighting prompts to help trainees quickly locate the operation area, the auditory channel can provide real-time reminders through voice broadcasts, and the text channel can supplement the operation with detailed explanations. The collaborative feedback of these three channels not only meets the different receiving habits of trainees but also ensures the comprehensive delivery of guidance information, achieving dynamic guidance for trainees' operations. This helps trainees quickly grasp the core logic and standardized procedures of clinical nursing operations, improving the teaching effectiveness of virtual training.

[0073] The above method, firstly, effectively addresses the problems of traditional training lacking structured clinical decision-making logic support and rigid, fixed guidance rules by constructing a nursing clinical decision-making knowledge graph with exclusive attribute edges. Secondly, it collects multimodal data and generates real-time patient state vectors through cross-modal fusion, compensating for the inadequacy of single-modal data in comprehensively reflecting patient states and improving the accuracy of state perception. Finally, it calculates decision weights based on activation subgraphs and graph attention networks, combined with multi-channel real-time feedback, effectively solving the problems of weak targeting and single-dimensional interaction in traditional guidance, enabling trainees to quickly master operational procedures and improving the teaching effectiveness and practicality of virtual training.

[0074] In one embodiment, a nursing clinical decision-making knowledge graph is constructed, comprising state nodes, operation nodes, and directed edges connecting state nodes and operation nodes; a semantic embedding vector is generated for each state node and each operation node, and a metadata embedding vector is generated for each operation node, including:

[0075] Knowledge data is collected from pre-set clinical nursing guidelines, expert experience knowledge base and historical nursing cases. A biomedical entity recognition model is used to identify state description entities and operation description entities in the knowledge data to obtain the initial state node set and the initial operation node set.

[0076] A remote supervision method combined with predefined rule templates is used to extract state-operation relationships and operation-state relationships between entities in the initial set of state nodes and the initial set of operation nodes, forming the initial first-type edge set and the initial second-type edge set, respectively. Semantic clustering and merging of synonymous entities in the initial set of state nodes and the initial set of operation nodes are performed to obtain the final set of state nodes and the final set of operation nodes. Based on the entity merging results, the relationship pointers in the initial first-type edge set and the initial second-type edge set are adjusted synchronously to obtain the final first-type edge set and the second-type edge set.

[0077] Based on pre-defined clinical nursing guidelines and expert experience knowledge base, the clinical effectiveness, execution priority, and risk level of the state-operation relationship corresponding to each edge in the first type of edge set are quantitatively scored to obtain the clinical expert score results for each first type of edge; according to the clinical expert score results, the corresponding expert rule strength is determined for each first type of edge in the first type of edge set; according to the pre-defined precondition description in the predefined rule template, the precondition description is converted into vector form through a natural language processing model to generate the operation precondition vector corresponding to each first type of edge in the first type of edge set;

[0078] Based on historical nursing case data, statistical analysis was performed to calculate the frequency of occurrence of the operation-state association relationship corresponding to each second-type edge in the second-type edge set in clinical practice, and the frequency of occurrence was used as the transition probability corresponding to each second-type edge. According to the preset clinical outcome assessment criteria, the clinical prognostic effect of the state node corresponding to each second-type edge in the second-type edge set was graded and scored, and the graded score was used as the expected outcome score corresponding to each second-type edge.

[0079] Based on the set of state nodes and the set of operation nodes, the descriptive text of each state node and each operation node is feature-encoded using a pre-trained medical BERT model to generate semantic embedding vectors for each state node and each operation node; the metadata of each operation node is feature-encoded to generate metadata embedding vectors for each operation node.

[0080] Using historical nursing case data, each historical nursing case is formatted as a state-operation transition sequence. By maximizing the log-likelihood function of expert decisions, the weights of each first-class edge in the first-class edge set are initially trained to obtain the trainable edge weights of each first-class edge.

[0081] Specifically, clinical nursing guidelines contain structured and unstructured texts such as disease nursing pathways and operational procedures. Expert experience knowledge bases store standardized clinical expert diagnostic and treatment suggestions and operational insights. Historical nursing cases cover real clinical practice data such as electronic medical records and nursing records. Knowledge data can be collected from these data using methods adapted to their data formats and storage characteristics. For example, if the structured text in the clinical nursing guidelines is already stored in a structured database, the target data can be exported in batches using a database query interface, based on preset fields such as disease name, operation name, and applicable scenario. Unstructured text can be collected using a combination of text extraction and preprocessing. First, a document parsing tool can be used to extract the plain text content from the guidelines, removing redundant information such as headers, footers, page numbers, and watermarks. Then, natural language processing technology can be used for sentence segmentation and deduplication, retaining core text fragments related to patient status descriptions and nursing operations to form a standardized unstructured text dataset. Furthermore, a biomedical entity recognition model fine-tuned for the clinical domain can be used to process the collected knowledge data. This model can be built on the BioBERT pre-trained architecture and fine-tuned by introducing a clinical entity annotation dataset (containing annotated samples of state description entities and operation description entities). This enables it to accurately identify entities in the text that represent the patient's clinical state (e.g., high fever) and entities that represent nursing operations (e.g., intravenous infusion). Illustratively, during the recognition process, the model can first perform word segmentation, part-of-speech tagging, and syntactic analysis on the text, and then output an initial set of state nodes and an initial set of operation nodes through entity boundary detection and type classification. Each element in the set corresponds to a unique entity identifier and original text description.

[0082] Specifically, predefined rule templates can be designed based on clinical nursing logic, including state-operation trigger templates and operation-state transition templates. Remote supervision methods can automatically label potential relational instances by matching rule templates with knowledge data, and then use these instances to train a relation extraction model. This allows for the extraction of state-operation and operation-state relations between entities in the initial set of state nodes and the initial set of operation nodes, forming the initial first-class edge set and the initial second-class edge set, respectively. For synonymous entities in the initial node set, such as "high fever" and "body temperature exceeding 38.5℃," a semantic clustering algorithm can be used for merging. First, the cosine similarity of the semantic embedding vectors of each entity is calculated, a similarity threshold is set, and entities with similarity higher than the threshold are grouped into one class. The most representative entity description in each class is selected as the merged node name, resulting in the final set of state nodes and the final set of operation nodes. Based on the entity merging results, the start and end point identifiers of the relations in the initial first-class edge set and the initial second-class edge set can be further updated, and duplicate relations can be deleted, resulting in the final first-class edge set and the second-class edge set.

[0083] Specifically, for the expert rule strength of the first type of edge, at least three clinical nursing experts can independently quantify the clinical effectiveness, execution priority, and risk level of each state-operation relationship based on pre-defined clinical nursing guidelines and an expert experience knowledge base. The clinical effectiveness score reflects the applicability and effectiveness of the relationship in clinical practice; the execution priority score reflects the urgency and recommended order of the operation in the current state; and the risk level score characterizes the potential adverse consequences during operation execution. Based on these scores, a weighted summation method can be used to calculate the clinical expert score results, where the weights can be determined using the Delphi method. For example, clinical effectiveness can have the highest weight, followed by execution priority, and risk level can have the lowest weight. Finally, the result can be normalized and used as the expert rule strength of the corresponding first type of edge. For the operational prerequisite vector of the first type of edge, the pre-defined prerequisite descriptions, such as the patient being conscious, can be extracted from the predefined rule template. These text descriptions can be converted into vector form by a natural language processing model. This model can adopt the RoBERTa architecture. After fine-tuning with a clinical prerequisite text dataset, it can map the natural language descriptions to a fixed-dimensional vector space. The generated operational prerequisite vectors can be directly used for subsequent adaptation calculations.

[0084] Specifically, based on historical nursing case data, the frequency of a specific subsequent state occurring after performing a certain operation in historical cases can be counted. Dividing this frequency by the total number of times the operation was performed in all cases yields the transition probability of the corresponding second-type edge. This probability reflects the degree of influence of the operation on state transition. The expected outcome score can be determined based on preset clinical outcome assessment criteria. These criteria, combining disease prognostic indicators and symptom improvement, classify clinical prognostic effects into multiple levels. Subsequently, clinical experts can be organized to grade and score the target state node corresponding to each second-type edge. The score results, after normalization, can be used as the expected outcome score for that second-type edge, quantifying the clinical value brought by the operation.

[0085] Furthermore, based on the final set of state nodes and operation nodes, a pre-trained medical BERT model can be used for feature encoding. This model can use PubMed summaries and clinical electronic medical record data as pre-training corpora, and optimize for the semantic features of biomedical texts, thereby effectively capturing the professional meaning of clinical terms. For example, for each state node and operation node, its descriptive text can be input into the model. After processing through word segmentation, word embedding, multi-head self-attention mechanism, and feedforward neural network, the output vector at the [CLS] position of the model can be extracted as the semantic embedding vector of the node. This vector can accurately represent the semantic information of the node. For the metadata of the operation node, this metadata can contain structured information such as operation duration, required instruments, execution site, and contraindications. Therefore, the structured data can be converted into text descriptions first, and then concatenated with the semantic embedding vector. Through linear transformation, it can be mapped to a preset dimension to obtain the metadata embedding vector of the operation node. This vector supplements the technical details of the operation and provides support for subsequent suitability judgment.

[0086] Furthermore, based on historical nursing case data, trainable edge weights can be assigned to the first type of edges. For example, each historical nursing case can be formatted as a state-operation transition sequence, with the sequence format as follows: ,in Represents a state node. This represents the operation node, and the sequence is arranged in clinical time order. Based on this transition sequence, the edge weights are initially trained by maximizing the log-likelihood function of expert decisions. The core formula is:

[0087]

[0088] In the formula, The value is the log-likelihood function value. The set of state-operation transition triples for all historical cases. Indicates in the state node Select operation node below The probability, State Node Point to the operation node The first type of trainable edge weights State Node With operation node Cosine similarity of semantic embedding vectors State Node The set of all triggerable operation nodes. The training process can use the stochastic gradient descent algorithm to minimize the negative log-likelihood function, and iteratively update the trainable edge weights to make the operation selection probability output by the model fit the expert decision in historical cases as closely as possible. When the number of iterations reaches a preset value or the loss function converges, training stops, and the initial trainable edge weights of each first-class edge are obtained.

[0089] In one embodiment, features are extracted from the multimodal data to obtain corresponding single-modal feature vectors; each single-modal feature vector is input into a multimodal fusion encoder for cross-modal interactive fusion to generate a patient's real-time state vector, including:

[0090] The vital signs values ​​are normalized to obtain normalized vital signs values. The normalized vital signs values ​​are then input into a multilayer perceptron network for feature extraction to generate vital signs feature vectors. The facial expression images are standardized to obtain normalized facial expression images. The standardized facial expression images are then input into a pre-defined convolutional neural network for feature extraction to generate facial expression feature vectors.

[0091] The audio of the main complaint is segmented to obtain the segmented audio of the main complaint. The segmented audio of the main complaint is converted into text data through automatic speech recognition technology. Feature extraction is performed on the text data to generate the feature vector of the main complaint.

[0092] The local sign image is standardized, and the standardized local sign image is input into the semantic segmentation network for segmentation. Quantitative features including the proportion of red and swollen areas, the proportion of oozing areas, the probability of oozing color, and the shape factor are extracted from the segmentation results to generate a local sign feature vector.

[0093] The initial feature sequence is constructed by using the feature vectors of vital signs, facial expression, speech complaints, and local signs as sequence elements; modality type encoding and temporal location encoding are added to each element of the initial feature sequence to obtain the encoded feature sequence.

[0094] The encoded feature sequence is input into a multimodal fusion encoder based on the Transformer architecture. Information is exchanged and fused through multiple cross-modal self-attention layers, and the fused feature vectors of each modality are output. The mean of each feature vector is taken as the patient's real-time state vector.

[0095] Specifically, vital sign values ​​are continuous physiological data. Different types of vital sign values ​​have differences in dimensions and numerical ranges. Directly inputting them into the network can lead to an imbalance in feature distribution. Therefore, vital sign values ​​can be normalized first, mapping the original values ​​to a uniform numerical range and eliminating the impact of dimensional differences on subsequent feature extraction. The normalized vital sign values ​​are then input into a multilayer perceptron network, which can consist of an input layer, multiple hidden layers, and an output layer. The hidden layers use non-linear activation functions to achieve non-linear transformation of features. By performing feature mapping and abstraction on the normalized vital sign values ​​layer by layer, high-dimensional features that can represent the patient's physiological state can be extracted, ultimately outputting a fixed-dimensional vital sign feature vector. Facial expression images can be used to reflect the patient's subjective state such as pain and discomfort, but the original images have problems such as inconsistent size and uneven pixel distribution. Therefore, facial expression images can be standardized first, unifying the image size and pixel value range to ensure the consistency of input data. The standardized facial expression image is input into a pre-defined convolutional neural network. This network can extract local features such as facial texture, contour, and muscle deformation through convolutional layers, reduce feature dimensionality and preserve key information through pooling layers, and integrate local features into global features through fully connected layers, ultimately generating a facial expression feature vector that can represent the patient's facial expression state.

[0096] Specifically, the audio of the patient's subjective complaint includes descriptions of symptoms and subjective requests. Continuous audio can be segmented into segments based on silence detection results or fixed durations to avoid feature extraction distortion caused by long audio clips. Then, automatic speech recognition technology can convert the segmented audio into text data. This technology, based on acoustic and language models, achieves accurate audio-to-text conversion, and the model can be pre-tuned using clinical nursing corpora to accurately identify medical terminology and symptom descriptions. Feature encoding is then performed on the converted text data to extract semantic information, generating a voice complaint feature vector that represents the content of the patient's complaint.

[0097] Furthermore, local physical sign images can be used to reflect the objective state of the patient's lesion site. Standardizing these images first unifies image specifications and pixel distribution. The standardized images are then input into a semantic segmentation network, which can accurately segment lesion areas such as swollen and effusion areas. Based on the segmentation results, quantitative features such as the proportion of swollen areas, the proportion of effusion areas, the probability of effusion color, and shape factors can be extracted. These quantitative features are then concatenated and mapped to generate a local physical sign feature vector that comprehensively characterizes the patient's local lesion state.

[0098] Specifically, the generated vital sign feature vectors, facial expression feature vectors, speech complaint feature vectors, and local sign feature vectors are arranged in a preset order to form an initial feature sequence. Then, modality type encoding and temporal location encoding can be added to each feature vector in the initial feature sequence. Modality type encoding is used to distinguish the feature sources of different modalities, while temporal location encoding is used to represent the acquisition timing information corresponding to each feature vector. Therefore, the encoded feature sequence can simultaneously retain modality specificity and temporal correlation. This sequence is input into a multimodal fusion encoder based on the Transformer architecture. This encoder consists of multiple layers of cross-modal self-attention layers and a feedforward network. It can realize information interaction and global correlation modeling between features of different modalities through cross-modal self-attention mechanism, capture complementary information and dependencies between modalities, and output fused feature vectors for each modality. Finally, the mean of each modality feature vector can be calculated, and the calculation result is used as the patient's real-time state vector.

[0099] In one embodiment, such as Figure 2 As shown, the similarity between the patient's immediate state vector and the semantic embedding vectors of each state node in the nursing clinical decision-making knowledge graph is calculated. Based on the similarity results, the corresponding state nodes and neighboring nodes are activated to form an activation subgraph, including:

[0100] S201: Calculate the cosine similarity between the patient's immediate state vector and the semantic embedding vector of each state node in the nursing clinical decision knowledge graph, and obtain the similarity value of each state node.

[0101] S202: Select state nodes with similarity values ​​higher than the preset similarity threshold as activation nodes; if the similarity values ​​of all state nodes are lower than the preset similarity threshold, then arrange the state nodes according to their similarity values ​​and select the first preset number of state nodes as activation nodes.

[0102] S203: Extract the state nodes, operation nodes and corresponding directed edges in the two-hop neighborhood of each activated node, and integrate them with the activated nodes to form an activated subgraph; wherein, the two-hop neighborhood includes the first-level neighborhood nodes directly connected to the activated node, and the second-level neighborhood nodes directly connected to the first-level neighborhood nodes.

[0103] Specifically, based on the semantic matching principle of the vector space model, a cosine similarity algorithm can be used to quantify the correlation between the patient's immediate state vector and the semantic embedding vectors of each state node. This algorithm can characterize the consistency of vector directions by calculating the cosine of the angle between two vectors; a larger value indicates a stronger semantic correlation. Furthermore, before calculation, the patient's immediate state vector and the semantic embedding vectors of the state nodes must have completely identical dimensions, both being preset fixed dimensions, to ensure the validity of the calculation. For example, the corresponding calculation formula can be:

[0104]

[0105] In the formula, The cosine similarity value is between the patient's instantaneous state vector and the semantic embedding vector of the target state node. For the patient's immediate state vector, This is the semantic embedding vector of a state node in the knowledge graph. This represents the transpose of the patient's instantaneous state vector. Representing vectors The norm is used to normalize vector length, eliminating the impact of vector scale differences on similarity calculation. This formula is used to calculate the similarity value for each state node by individually calculating the semantic embedding vector of all state nodes in the knowledge graph.

[0106] Specifically, the preset similarity threshold can be determined through offline experiments. This involves using cross-validation to select the optimal threshold based on the matching results of historical multimodal patient data and corresponding state nodes. This threshold maximizes the recall and precision of relevant state nodes. The similarity value of each state node is compared with the preset similarity threshold. State nodes with similarity values ​​higher than the threshold are selected as activation nodes, representing clinical states highly relevant to the current patient's immediate state. If the similarity values ​​of all state nodes are lower than the preset similarity threshold, it indicates that there is no directly matching standard state node for the current patient's state. In this case, all state nodes can be sorted from highest to lowest similarity value, and the top preset number of state nodes are selected as activation nodes. The preset number can be set considering both computational efficiency and decision comprehensiveness to ensure that neither potentially relevant states are overlooked nor too many irrelevant nodes are introduced, leading to computational redundancy.

[0107] Furthermore, after the activated nodes are determined, the two-hop neighborhood nodes and corresponding directed edges of each activated node can be extracted based on the topological structure of the knowledge graph. In the two-hop neighborhood, the first-hop neighborhood can cover the nursing operation nodes directly triggered by the activated node or the state nodes directly transferred, while the second-hop neighborhood can extend further to the subsequent state nodes corresponding to the operation nodes or other associated operation nodes corresponding to the state nodes. This ensures that the subgraph contains a complete state-operation-state decision chain while avoiding excessive expansion of the subgraph size due to an overly large neighborhood range. For example, the adjacency list of each activated node can be traversed firstly to obtain its directly connected first-level neighborhood nodes (including operation nodes connected via first-type edges and state nodes connected via second-type edges) and their corresponding directed edges. Then, the adjacency list of each first-level neighborhood node can be traversed to obtain its directly connected second-level neighborhood nodes and their corresponding directed edges. If the first-level node is an operation node, the second-level node is a state node connected via a second-type edge; if the first-level node is a state node, the second-level node is an operation node connected via a first-type edge. Finally, all extracted nodes and directed edges can be integrated, removing duplicate nodes and edges to ensure the activation subgraph has a concise and logically complete topological structure. The integrated node set can contain activation nodes, first-level neighbor nodes, and second-level neighbor nodes, while the edge set can contain first-type and second-type edges between nodes. Ultimately, this forms the activation subgraph, providing a data foundation for subsequent decision calculations.

[0108] In one embodiment, features of each node in the activation subgraph are initialized based on the patient's immediate state vector and the semantic embedding vector of the corresponding state node in the nursing clinical decision knowledge graph; the activation subgraph is input into a graph attention network with trainable parameters for multi-layer information propagation, the decision weight of each operation node is calculated, operation nodes that meet preset conditions based on their decision weights are selected, and multi-channel guidance instructions are generated, including:

[0109] The nodes in the activation subgraph are initialized with features to obtain the initialized node features;

[0110] Extract the attribute vectors of each directed edge in the activation subgraph, input the initialized node features and the attribute vectors of each directed edge in the activation subgraph into the graph attention network, calculate the attention coefficient between each node and its corresponding neighbor nodes in the activation subgraph through a multi-head attention mechanism, and fuse the attribute vectors of the corresponding directed edges in the process of calculating the attention coefficients to obtain the final attention coefficients.

[0111] Based on the final attention coefficients, multi-layer node update processing is performed in the graph attention network, updating the feature representation of each node in the activation subgraph layer by layer, to obtain the final feature representation of each operation node in the activation subgraph and the average attention coefficients passed from each activation node in the activation subgraph to the operation node.

[0112] For each operation node in the activation subgraph, the precondition matching degree of the operation node in the activation subgraph is calculated by a preset multilayer perceptron based on the patient's immediate state vector and the corresponding semantic embedding vector. Based on the transition probability and expected outcome score of each next state node connected by the operation node in the activation subgraph through the second type of edge, and combined with the similarity between the patient's immediate state vector and the semantic embedding vector of each next state node, the neighbor expected score of the operation node in the activation subgraph is calculated.

[0113] The mean attention coefficient, the degree of matching of preconditions, and the expected score of neighbors are weighted and summed according to a preset weight ratio to obtain the decision weight of each operation node in the activation subgraph.

[0114] The operation nodes in the activation subgraph are arranged from high to low according to the decision weight. The top K operation nodes with decision weights higher than the preset weight threshold are selected to obtain the filtered operation nodes. The visual highlighting instructions, voice broadcast text and text prompt information corresponding to the filtered operation nodes are generated through the preset mapping table. The visual highlighting instructions, voice broadcast text and text prompt information are combined into multi-channel guidance instructions.

[0115] The following steps are involved in feature initialization:

[0116] For a state node in the activation subgraph, if the state node in the activation subgraph belongs to the activation node, the corresponding feature is initialized as the patient's immediate state vector; if the state node in the activation subgraph does not belong to the activation node, the corresponding feature is initialized as the semantic embedding vector of the state node itself in the activation subgraph.

[0117] For an operation node in the activation subgraph, the semantic embedding vector of the operation node in the activation subgraph is concatenated with the corresponding metadata embedding vector, and the concatenated vector is mapped to a preset dimension through a linear transformation to obtain the initial features of the operation node in the activation subgraph.

[0118] Specifically, when initializing features for nodes in the activation subgraph, if a state node in the activation subgraph is an active node, since the active node is highly correlated with the current patient state, its corresponding features can be initialized as the patient's immediate state vector. This ensures that the decision-making process closely aligns with the actual state of the current training scenario, rather than relying on the node's general semantics. If the state node is not an active node, its features can be initialized as its own semantic embedding vector, preserving the node's general clinical semantic information. For operation nodes in the activation subgraph, the corresponding semantic embedding vector represents the operation's clinical semantic information, while the corresponding metadata embedding vector supplements the operation's technical parameters, execution requirements, and other details. Therefore, the operation node's semantic embedding vector and the corresponding metadata embedding vector can be concatenated. The concatenated vector can be mapped to a preset dimension through a linear transformation, thus obtaining the operation node's initial features. This ensures that the operation node's initial features simultaneously consider semantic adaptability and technical detail adaptability.

[0119] Subsequently, the attribute vectors of each directed edge in the activation subgraph can be extracted, and the initialized node features and edge attribute vectors are input into the graph attention network. This graph attention network adopts a multi-head attention mechanism, which can capture node association information of different dimensions in parallel through multi-head attention, and can integrate edge attribute vectors in the attention coefficient calculation to reflect the clinical significance of edges in the nursing clinical decision knowledge graph. Illustratively, the original single-head attention coefficient can be calculated first; for example, its formula is:

[0120]

[0121] Where k is the kth head of the multi-head attention, and K is the total number of multi-head attention; Let be the single-head original attention coefficient of node i in the k-th head to its neighbor node j; Let be the attention weight vector for the k-th head; Let be the linear transformation matrix of the node features of the k-th head; These are the initial feature vectors for nodes i and j, respectively. Let be the attribute vector of the directed edge from node i to node j; Let be the linear transformation matrix of the edge attribute features of the k-th head; Let i be the set of neighboring nodes of node i. This formula combines the features of node i, the features of node j, and the edges... After concatenating the attribute vectors, the attention weight vector is then applied. The mapping is then performed, followed by the introduction of nonlinearity through the LeakyReLU activation function, and finally normalized using Softmax to obtain the single-head attention coefficients, thus quantifying the association strength between nodes. Subsequently, the average of the original single-head attention coefficients of the K heads can be taken to obtain the average multi-head attention coefficients, as shown in the formula: This mean incorporates multi-dimensional correlation information captured by multi-head attention, thus improving the robustness of the attention coefficient.

[0122] Based on the above average multi-head attention coefficient The L2 norm of the edge attribute vector can be normalized using the Sigmoid activation function to correct the mean of the multi-head attention coefficients, thus obtaining the final attention coefficients. The formula is as follows:

[0123]

[0124] in, Use the Sigmoid activation function; Let L2 norm be the edge attribute vector; This is the maximum value of the L2 norm of all neighboring attribute vectors of node i. This correction process adjusts the attention coefficient based on the importance of edge attributes (characterized by the L2 norm), enabling edges with greater clinical significance to have a higher weight in information dissemination.

[0125] Furthermore, based on the final attention coefficients, multi-layer node update processing can be performed in the graph attention network. In each layer update, the feature of node i is obtained by weighted aggregation of its own features and the features of its neighboring nodes through the final attention coefficients. After layer-by-layer iteration, the feature representations of each node in the activation subgraph gradually fuse with neighborhood information, and finally the final feature representation of each operating node can be obtained. At the same time, the average attention coefficients passed from the activation node to the operating node can also be calculated, which represents the overall attention level of the activation node to the operating node.

[0126] Specifically, when calculating the precondition matching degree of each operation node in the activation subgraph, the patient's immediate state vector and the operation precondition vector corresponding to the operation node can be input into a preset multilayer perceptron. This multilayer perceptron can consist of multiple fully connected layers and activation functions, and can capture the complex matching relationship between the patient's state and the operation preconditions through nonlinear transformations, outputting the precondition matching degree of the operation node. The larger this value, the more the patient's state meets the operation's precondition requirements. When calculating the neighbor expectation score, it is possible to traverse each next-state node connected to the operation node by the second type of edge. For each next-state node, the product of the transition probability and the expected outcome score is calculated, and then multiplied by the cosine similarity between the patient's immediate state vector and the semantic embedding vector of the next-state node. The calculation results of all next-state nodes are summed to obtain the neighbor expectation score of the operation node. This score integrates the probability of state transition after operation, the clinical value of the transitioned state, and the adaptability to the current patient state, quantifying the expected clinical benefits brought by the operation.

[0127] Specifically, the decision weight of each operation node can be obtained by weighting and summing the mean attention coefficient, the degree of matching of preconditions, and the neighbor expectation score according to a preset weight ratio. This preset weight can be determined through offline experiments and expert experience to ensure that the contribution of each factor to the decision aligns with clinical logic. By sorting the operation nodes from highest to lowest decision weight, the top K operation nodes with decision weights exceeding the preset weight threshold can be selected. These nodes represent the most recommended nursing operations in the current scenario. Finally, a preset mapping table can be used to map the selected operation nodes to corresponding visual highlighting instructions, voice broadcast text, and text prompts. Visual highlighting instructions can be used to highlight the operation area in the virtual training scenario, helping trainees quickly locate the operation target. Voice broadcast text can instantly broadcast key points of the operation through the auditory channel, providing real-time reminders. Text prompts can supplement the clinical basis and precautions of the operation through the text channel. Therefore, combining these three into a multi-channel guidance instruction allows for multi-channel collaborative feedback, meeting the information reception habits of different trainees and improving the teaching effectiveness of virtual training.

[0128] In one embodiment, the method further includes:

[0129] During the process of trainees performing nursing operations based on multi-channel guided instructions, the real-time status vectors of patients at each time step are collected according to the preset time step, arranged in chronological order, and a status sequence is generated.

[0130] Record the operation node index corresponding to the nursing operation actually performed by the trainee in the virtual training scenario at each time step, arrange them in chronological order, and generate an operation sequence;

[0131] Based on graph attention network, the state sequence is processed step by step, the decision weight of the corresponding operation node at each time step is calculated, and the decision weight sequence is generated by arranging them in chronological order. Based on the matching degree between the virtual patient's state change results after the trainee performs nursing operations based on multi-channel guidance instructions at each time step and the preset standard state change, the instant reward for the corresponding time step is calculated, and the instant reward sequence is generated by arranging them in chronological order.

[0132] After performing nursing procedures at each time step, the corresponding outcome scores are calculated based on the virtual patient's final vital signs, local signs, and pain level. These scores are then arranged in chronological order to obtain the outcome score sequence.

[0133] The state sequence, operation sequence, decision weight sequence, immediate reward sequence, and outcome score sequence are stored to construct a training trajectory database;

[0134] When the total number of trajectories accumulated in the training trajectory database reaches a preset threshold, a preset batch of four-tuple data including state, operation, reward and next state is sampled from the training trajectory database.

[0135] A Q-network with the same structure as the graph attention network is constructed. The conservative Q-learning algorithm is used to update the network parameters of the Q-network by minimizing the Bellman error, and the updated Q-network is obtained.

[0136] Using the graph attention network as the policy network, the network parameters of the policy network are updated by minimizing the KL divergence between the updated Q network and the policy network, thus obtaining the updated policy network. The Q value output by the updated Q network is normalized, and the trainable edge weights of the first type of edge in the nursing clinical decision knowledge graph are adjusted by the momentum update method, thus obtaining the updated trainable edge weights.

[0137] Specifically, during the process of trainees performing nursing operations based on multi-channel guided instructions, the system can collect the patient's real-time state vectors corresponding to each time step according to a preset time step. This preset time step can be set according to the operational rhythm of the virtual training scenario, such as dividing it by operation step nodes or fixed time intervals, ensuring that the collected state sequence can fully reflect the dynamic changes in the patient's state as the operation progresses. Arranging the collected patient real-time state vectors in chronological order generates a state sequence. Furthermore, the system can record the operation node indices corresponding to the nursing operations actually performed by the trainee in the virtual training scenario at each time step. These operation node indices correspond one-to-one with the operation nodes in the knowledge graph, and arranging them in chronological order generates an operation sequence that accurately reconstructs the trainee's operational decision-making path.

[0138] Subsequently, the state sequence can be processed step-by-step using a graph attention network. The patient's immediate state vector at each time step is input into the graph attention network, and the decision weights of all operation nodes at that time step are calculated. These decision weights are then arranged chronologically to generate a decision weight sequence, which can be used to compare the fit between the trainee's actual operations and the system's recommended operations. Furthermore, immediate rewards can be calculated based on the matching degree between the virtual patient's state changes after the trainee performs nursing operations at each time step and the preset standard state changes. The preset standard state changes can be defined by clinical nursing guidelines, and the matching degree can be obtained by calculating the cosine similarity between the actual state change vector and the standard state change vector. Higher similarity results in higher immediate rewards, and vice versa. Arranging the calculated immediate rewards chronologically yields an immediate reward sequence, representing the immediate clinical effect of each operation. After performing nursing procedures at each time step, an outcome score can be calculated by weighted summation based on the virtual patient's final vital signs, local signs, and pain level. Vital signs can be quantified by deviations from indicators such as blood pressure and heart rate; local signs can be assessed by the degree of improvement in lesion features such as redness, swelling, and effusion; and pain level can be determined by a combination of facial expressions and vocal characteristics. The resulting outcome scores are arranged chronologically, and the resulting sequence can be used to characterize the long-term clinical benefits of each procedure. Finally, the state sequence, procedure sequence, decision weight sequence, immediate reward sequence, and outcome score sequence can be stored in a structured format to construct a training trajectory database. Each trajectory in the database corresponds to a complete virtual training process, providing data support for subsequent offline reinforcement learning.

[0139] As an illustration, when the total number of trajectories accumulated in the training trajectory database reaches a preset threshold, a preset batch of four-tuple data can be sampled from the database through uniform sampling or preferential sampling. This four-tuple includes the current state, the action to be performed, the immediate reward, and the next state. Preferential sampling of high-reward trajectories can further improve learning efficiency, while the preset batch size can be determined comprehensively based on computational resources and training efficiency. Subsequently, a Q-network with a structure consistent with the graph attention network can be constructed. The input of the Q-network is the state vector and the features of the operation nodes, and the output is the value estimate of the corresponding action. The conservative Q-learning algorithm is used to update the Q-network parameters, i.e., to minimize the Bellman error, and a conservative term is introduced to suppress overestimation of the Q-value. Its core formula can be:

[0140]

[0141] In the formula, Let Q be the loss function of the Q network. This is the dataset of sampled quadruplets. Let this be the current state vector. For the operation node to be executed, For instant rewards, The next state vector, This is a discount factor used to balance the weights of immediate and future rewards. For Q-network set, For the policy network in state The distribution of actions below, As a conservative measure, overestimation is suppressed by taking the minimum value of multiple Q-network outputs. This loss function is minimized using the gradient descent algorithm, and the parameters of the Q-network are iteratively updated to obtain the updated Q-network, which can more accurately estimate the long-term value of actions.

[0142] Specifically, the graph attention network is used as the policy network, and the output of the policy network is the distribution of decision weights for the operation nodes. Since KL divergence can be used to measure the difference between two probability distributions, the policy network parameters can be updated by minimizing the KL divergence between the updated Q network and the policy network. This makes the action distribution of the policy network as close as possible to the operation with the higher Q value, improving the effectiveness of the policy. During the update process, the parameters of the policy network can be iteratively optimized using the gradient descent algorithm until the KL divergence converges, resulting in an updated policy network whose decision logic is more consistent with effective decisions in offline trajectories. Furthermore, the Q value output by the updated Q network can be normalized to map the Q value to a uniform interval, eliminating the impact of numerical fluctuations on edge weight updates. Based on the normalized result, the trainable edge weights of the first type of edge in the nursing clinical decision knowledge graph can be adjusted using a momentum update method. For example, its formula can be:

[0143]

[0144] In the formula, For the updated first type of edge (state node) Point to the operation node The trainable edge weights of ) The trainable edge weights before the update. The momentum coefficient controls the retention ratio of historical weights. The normalized Q value, State Node A set of triggerable operation nodes. This formula allows for the updating of trainable edge weights, preserving historical experience while incorporating new action value information obtained from offline learning. This enables dynamic optimization of the knowledge graph's edge weights, making them more aligned with real clinical decision-making logic.

[0145] Based on the same inventive concept, this application also provides a nursing operation virtual training intelligent teaching system for implementing the above-mentioned nursing operation virtual training intelligent teaching method. The solution provided by this system is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the nursing operation virtual training intelligent teaching system provided below can be found in the limitations of the nursing operation virtual training intelligent teaching method above, and will not be repeated here.

[0146] In one exemplary embodiment, such as Figure 3 As shown, a nursing operation virtual training intelligent teaching system 300 is provided, including:

[0147] The graph construction module 301 is used to construct a nursing clinical decision knowledge graph including state nodes, operation nodes, and directed edges connecting state nodes and operation nodes; it generates semantic embedding vectors for each state node and each operation node, and generates metadata embedding vectors for each operation node; wherein, the directed edges include first-type edges pointing from state nodes to operation nodes and second-type edges pointing from operation nodes to state nodes, the first-type edge attributes include expert rule strength, trainable edge weights, and operation precondition vectors, and the second-type edge attributes include transition probability and expected outcome score;

[0148] The state perception module 302 is used to collect multimodal data of the digital twin patient in the virtual training scene in real time. The multimodal data includes vital sign values, facial expression images, voice complaints audio, and local vital sign images. Features are extracted from the data in the multimodal data to obtain the corresponding single-modal feature vectors. The single-modal feature vectors are input into the multimodal fusion encoder for cross-modal interactive fusion to generate the patient's real-time state vector.

[0149] The decision-making guidance module 303 is used to calculate the similarity between the patient's immediate state vector and the semantic embedding vectors of each state node in the nursing clinical decision knowledge graph. Based on the similarity results, the corresponding state node and its neighboring nodes are activated to form an activation subgraph. Based on the patient's immediate state vector and the semantic embedding vectors of the corresponding state nodes in the nursing clinical decision knowledge graph, the features of each node in the activation subgraph are initialized. The activation subgraph is input into a graph attention network with trainable parameters for multi-layer information propagation, and the decision weight of each operation node is calculated. Operation nodes that meet the preset conditions based on the decision weight are selected, and multi-channel guidance instructions are generated. The multi-channel guidance instructions are given to the trainees in real time through visual, auditory, and text channels. The multi-channel guidance instructions are used to instruct the trainees to perform corresponding nursing operations in the virtual training scenario.

[0150] In one exemplary embodiment, the present invention also provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the intelligent teaching method for virtual training of nursing operations according to this application. A multi-core processor is preferred to improve the system's parallel processing capability. The memory provides sufficient temporary storage space to support program execution and data processing. The memory capacity should be large enough to accommodate large amounts of data and computational tasks.

[0151] In one exemplary embodiment, the present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the intelligent teaching method for virtual training of nursing operations according to this application. The computer-readable storage medium may include: a read-only memory, a random access memory, a solid-state drive, or an optical disk, etc.

[0152] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A virtual training intelligent teaching method for nursing operations, characterized in that, The method includes: A nursing clinical decision-making knowledge graph is constructed, comprising state nodes, operation nodes, and directed edges connecting the state nodes and operation nodes. Semantic embedding vectors are generated for each state node and each operation node, and metadata embedding vectors are generated for each operation node. The directed edges include a first type of edge pointing from the state node to the operation node and a second type of edge pointing from the operation node to the state node. The first type of edge attributes include expert rule strength, trainable edge weight, and operation precondition vector. The second type of edge attributes include transition probability and expected outcome score. Multimodal data of a digital twin patient in a virtual training scenario is collected in real time. The multimodal data includes vital sign values, facial expression images, audio of the patient's voice complaints, and images of local vital signs. Feature extraction is performed on the data in the multimodal data to obtain corresponding single-modal feature vectors. The single-modal feature vectors are input into a multimodal fusion encoder for cross-modal interactive fusion to generate a real-time patient state vector. The similarity between the patient's immediate state vector and the semantic embedding vectors of each state node in the nursing clinical decision-making knowledge graph is calculated. Based on the similarity result, the corresponding state node and its neighboring nodes are activated to form an activation subgraph. The features of each node in the activation subgraph are initialized based on the patient's immediate state vector and the semantic embedding vectors of the corresponding state nodes in the nursing clinical decision-making knowledge graph. The activation subgraph is input into a graph attention network with trainable parameters for multi-layer information propagation. The decision weight of each operation node is calculated, and operation nodes that meet preset conditions based on their decision weights are selected to generate multi-channel guidance instructions. These multi-channel guidance instructions are then provided to the trainee in real time through visual, auditory, and text channels. These multi-channel guidance instructions instruct the trainee to perform corresponding nursing operations in a virtual training scenario.

2. The method according to claim 1, characterized in that, The construction includes a nursing clinical decision knowledge graph consisting of state nodes, operation nodes, and directed edges connecting the state nodes and operation nodes. Generate a semantic embedding vector for each state node and each operation node, and generate a metadata embedding vector for each operation node, including: Knowledge data is collected from preset clinical nursing guidelines, expert experience knowledge base and historical nursing cases. A biomedical entity recognition model is used to identify the state description entities and operation description entities in the knowledge data to obtain the initial state node set and the initial operation node set. Using a remote supervision method combined with a predefined rule template, the state-operation association and operation-state association between entities in the initial state node set and the initial operation node set are extracted to form the initial first type edge set and the initial second type edge set, respectively. Semantic clustering is performed on synonymous entities in the initial set of state nodes and the initial set of operation nodes to obtain the final set of state nodes and the final set of operation nodes. Based on the entity merging result, the association relationships in the initial first set of edges and the initial second set of edges are adjusted synchronously to obtain the final first set of edges and the final second set of edges. Based on the preset clinical nursing guidelines and the expert experience knowledge base, the clinical effectiveness, execution priority, and risk level of the state-operation association corresponding to each edge in the first type of edge set are quantitatively scored to obtain the clinical expert score results for each first type of edge; according to the clinical expert score results, the corresponding expert rule strength is determined for each first type of edge in the first type of edge set. Based on the predefined precondition description in the predefined rule template, the precondition description is converted into vector form by a natural language processing model to generate the operation precondition vector corresponding to each first-class edge in the first-class edge set; Based on the historical nursing case data, statistical analysis is performed to calculate the frequency of occurrence of the operation-state association relationship corresponding to each second type edge in the second type edge set in clinical practice, and the occurrence frequency is used as the transition probability corresponding to each second type edge; according to the preset clinical outcome evaluation criteria, the clinical prognostic effect of the state node corresponding to each second type edge in the second type edge set is graded and scored, and the graded score is used as the expected outcome score corresponding to each second type edge. Based on the set of state nodes and the set of operation nodes, a pre-trained medical BERT model is used to encode the descriptive text of each state node and each operation node to generate semantic embedding vectors for each state node and each operation node; and the metadata of each operation node is encoded to generate metadata embedding vectors for each operation node. Using the historical nursing case data, each historical nursing case is formatted as a state-operation transition sequence. By maximizing the log-likelihood function of expert decision, the weights of each first-class edge in the first-class edge set are initially trained to obtain the trainable edge weights of each first-class edge.

3. The method according to claim 1, characterized in that, The feature extraction is performed on the data in the multimodal data to obtain the corresponding single-modal feature vector; The single-modal feature vectors are input into a multimodal fusion encoder for cross-modal interactive fusion to generate a real-time patient state vector, including: The vital sign values ​​are normalized to obtain normalized vital sign values. The normalized vital sign values ​​are then input into a multilayer perceptron network for feature extraction to generate a vital sign feature vector. The facial expression image is standardized to obtain the standardized facial expression image, which is then input into a preset convolutional neural network for feature extraction to generate a facial expression feature vector. The audio of the main complaint is segmented to obtain segmented audio of the main complaint. The segmented audio of the main complaint is converted into text data by automatic speech recognition technology. Feature extraction is performed on the text data to generate a feature vector of the main complaint. The local sign image is standardized, and the standardized local sign image is input into a semantic segmentation network for segmentation. Quantitative features including the proportion of red and swollen areas, the proportion of oozing areas, the probability of oozing color, and the shape factor are extracted from the segmentation results to generate a local sign feature vector. The vital signs feature vector, the facial expression feature vector, the voice complaint feature vector, and the local vital signs feature vector are used as sequence elements to form an initial feature sequence; Add modality type encoding and temporal position encoding to each element in the initial feature sequence to obtain the encoded feature sequence; The encoded feature sequence is input into the multimodal fusion encoder based on the Transformer architecture. Information interaction and fusion are performed through multiple cross-modal self-attention layers, and the fused modal feature vectors are output. The mean of each modal feature vector is taken as the patient's real-time state vector.

4. The method according to claim 1, characterized in that, The step of calculating the similarity between the patient's immediate state vector and the semantic embedding vectors of each state node in the nursing clinical decision knowledge graph, and activating the corresponding state node and its neighboring nodes based on the similarity result to form an activation subgraph, includes: Calculate the cosine similarity between the patient's immediate state vector and the semantic embedding vector of each state node in the nursing clinical decision knowledge graph to obtain the similarity value of each state node. Select the state nodes whose similarity values ​​are higher than a preset similarity threshold as activation nodes; if the similarity values ​​of all the state nodes are lower than the preset similarity threshold, then arrange the state nodes according to their similarity values ​​and select the first preset number of state nodes as activation nodes. Extract the state nodes, operation nodes, and corresponding directed edges within the two-hop neighborhood of each activated node, and integrate them with the activated node to form the activated subgraph; wherein, the two-hop neighborhood includes the first-level neighborhood nodes directly connected to the activated node, and the second-level neighborhood nodes directly connected to the first-level neighborhood nodes.

5. The method according to claim 4, characterized in that, The process involves initializing the features of each node in the activation subgraph based on the patient's immediate state vector and the semantic embedding vector of the corresponding state node in the nursing clinical decision knowledge graph; inputting the activation subgraph into a graph attention network with trainable parameters for multi-layer information propagation; calculating the decision weight of each operation node; selecting operation nodes that meet preset conditions based on the decision weights; and generating multi-channel guidance instructions, including: The nodes in the activation subgraph are initialized with features to obtain initialized node features; Extract the attribute vectors of each directed edge in the activation subgraph, input the initialized node features and the attribute vectors of each directed edge in the activation subgraph into the graph attention network, calculate the attention coefficient between each node in the activation subgraph and its corresponding neighbor nodes through a multi-head attention mechanism, and fuse the attribute vectors of the corresponding directed edges during the calculation of the attention coefficients to obtain the final attention coefficients. Based on the final attention coefficients, multi-layer node update processing is performed in the graph attention network to update the feature representation of each node in the activation subgraph layer by layer, so as to obtain the final feature representation of each operation node in the activation subgraph and the average attention coefficients passed from each activation node in the activation subgraph to the operation node. For each operation node in the activation subgraph, the precondition matching degree of the operation node in the activation subgraph is calculated by a preset multilayer perceptron based on the patient's immediate state vector and the corresponding semantic embedding vector. Based on the transition probability and expected outcome score of each next state node connected by the operation node in the activation subgraph through the second type of edge, and combined with the similarity between the patient's immediate state vector and the semantic embedding vector of each next state node, the neighbor expected score of the operation node in the activation subgraph is calculated. The average attention coefficient, the precondition matching degree, and the neighbor expected score are weighted and summed according to a preset weight ratio to obtain the decision weight of each operation node in the activation subgraph. The operation nodes in the activation subgraph are arranged from high to low according to the decision weight. The top K operation nodes with decision weights higher than the preset weight threshold are selected to obtain the selected operation nodes. The visual highlighting instructions, voice broadcast text and text prompt information corresponding to the selected operation nodes are generated through the preset mapping table. The visual highlighting instructions, voice broadcast text and text prompt information are combined into the multi-channel guidance instructions. The following steps are involved in the feature initialization: For a state node in the activation subgraph, if the state node in the activation subgraph belongs to the activation node, the corresponding feature is initialized as the patient's immediate state vector; if the state node in the activation subgraph does not belong to the activation node, the corresponding feature is initialized as the semantic embedding vector of the state node itself in the activation subgraph. For the operation node in the activation subgraph, the semantic embedding vector of the operation node in the activation subgraph is concatenated with the corresponding metadata embedding vector, and the concatenated vector is mapped to a preset dimension through a linear transformation to obtain the initial features of the operation node in the activation subgraph.

6. The method according to claim 1, characterized in that, The method further includes: During the process of the trainee performing the nursing operation based on the multi-channel guidance command, the patient's real-time state vector corresponding to each time step is collected according to the preset time step, and arranged in chronological order to generate a state sequence. Record the operation node index corresponding to the nursing operation actually performed by the trainee in the virtual training scenario at each time step, arrange them in chronological order, and generate an operation sequence; Based on the graph attention network, the state sequence is processed step by step to calculate the decision weight of the operation node corresponding to each time step, and arranged in chronological order to generate a decision weight sequence; based on the matching degree between the virtual patient's state change result after the trainee performs the nursing operation based on the multi-channel guidance instruction and the preset standard state change at each time step, the instant reward corresponding to the time step is calculated and arranged in chronological order to generate an instant reward sequence. After performing the nursing operation at each time step, the outcome score corresponding to the nursing operation at each time step is calculated based on the final vital signs, local signs and pain level of the virtual patient. The scores are then arranged in chronological order to obtain the outcome score sequence. The state sequence, the operation sequence, the decision weight sequence, the instant reward sequence, and the outcome scoring sequence are stored to construct a training trajectory database; When the total number of trajectories accumulated in the training trajectory database reaches the preset number threshold, a preset batch of quadruple data including state, operation, reward and next state is sampled from the training trajectory database. A Q-network with the same structure as the graph attention network is constructed. The conservative Q-learning algorithm is used to update the network parameters of the Q-network by minimizing the Bellman error, and the updated Q-network is obtained. Using the graph attention network as the policy network, the network parameters of the policy network are updated by minimizing the KL divergence between the updated Q network and the policy network, thus obtaining the updated policy network. The Q-values ​​output by the updated Q-network are normalized, and the trainable edge weights of the first type of edge in the nursing clinical decision knowledge graph are adjusted using a momentum update method to obtain the updated trainable edge weights.

7. The method according to claim 5, characterized in that, The formula for calculating the final attention coefficient is as follows: in, The first point of focus for the bulls Size, This represents the total number of heads of multi-head attention. For the first Node in the head For neighboring nodes The single-head raw attention coefficient; For the first Attention weight vector for each size; For the first The linear transformation matrix of the node features of each head; They are nodes ,node Initialize the feature vector; For nodes To the node The attribute vector of the directed edge; For the first Linear transformation matrix of the edge attribute features of the head; For nodes The set of neighboring nodes; The average of the multi-head attention coefficients; Use the Sigmoid activation function; The final attention coefficient is given.

8. A virtual training intelligent teaching system for nursing operations, characterized in that, The system includes: The graph construction module is used to construct a nursing clinical decision-making knowledge graph including state nodes, operation nodes, and directed edges connecting the state nodes and operation nodes; it generates semantic embedding vectors for each state node and each operation node, and generates metadata embedding vectors for each operation node; wherein, the directed edges include a first type of edge pointing from the state node to the operation node and a second type of edge pointing from the operation node to the state node, the first type of edge attributes include expert rule strength, trainable edge weight, and operation precondition vector, and the second type of edge attributes include transition probability and expected outcome score; The state awareness module is used to collect multimodal data of a digital twin patient in a virtual training scenario in real time. The multimodal data includes vital sign values, facial expression images, audio of the patient's voice complaints, and images of local vital signs. Features are extracted from the data in the multimodal data to obtain corresponding single-modal feature vectors. The single-modal feature vectors are input into the multimodal fusion encoder for cross-modal interactive fusion to generate the patient's real-time state vector. The decision-guided teaching module is used to calculate the similarity between the patient's immediate state vector and the semantic embedding vectors of each state node in the nursing clinical decision knowledge graph, and activate the corresponding state node and its neighboring nodes based on the similarity results to form an activation subgraph. Based on the patient's immediate state vector and the semantic embedding vectors of the corresponding state nodes in the nursing clinical decision knowledge graph, the module initializes the features of each node in the activation subgraph. The activation subgraph is then input into a graph attention network with trainable parameters for multi-layer information propagation, calculating the decision weight of each operation node, selecting operation nodes that meet preset conditions based on the decision weights, and generating multi-channel guidance instructions. These multi-channel guidance instructions are then provided to the trainee in real time through visual, auditory, and text channels. The multi-channel guidance instructions instruct the trainee to perform corresponding nursing operations in a virtual training scenario.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images