Pediatric nursing practice teaching VR model generation method and system based on practical training video

Abstract

The invention discloses a pediatric nursing practice teaching VR model generation method and system based on a training video, and relates to the technical field of artificial intelligence and virtual reality, and the method comprises the steps: collecting a pediatric nursing operation video for preprocessing, extracting a key motion frame, separating medical staff, a child patient and a medical tool, converting the key motion frame into an editable 3D model, and carrying out the recognition of the editable 3D model; acquiring multi-modal data corresponding to the pediatric nursing operation, matching the converted 3D actions with semantics through a multi-modal large model, and optimizing 3D conversion; key contact points are marked in the 3D action sequence to carry out collision detection point burying, an operation process is defined to generate a 3D animation sequence with interaction logic to construct a VR teaching scene, the reasonability of the operation logic is verified through user testing, and scene correction is carried out. Through video analysis, 3D element replacement and animation point burying technologies, intelligent generation of pediatric nursing VR teaching resources is realized, and a more efficient and more immersive solution is provided for nursing education.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and virtual reality technology, and more specifically, to a method and system for generating a VR model for pediatric nursing practice teaching based on training videos. Background Art

[0002] Pediatric nursing practical teaching is a crucial component of medical education. Traditional training methods primarily rely on real-person simulations or video instruction. However, these methods have limitations: real-person simulations require significant human, material, and time resources and struggle to achieve high levels of repeatability; traditional video instruction lacks interactivity, preventing students from engaging in immersive practice, which impacts learning outcomes. In recent years, the application of virtual reality (VR) technology in medical education has become increasingly widespread, with its immersive and interactive nature providing a new solution for nursing skills training. However, existing VR teaching models typically rely on manual modeling and animation, resulting in long development cycles, high costs, and difficulty reproducing the details of real-world nursing scenarios. Furthermore, traditional 3D modeling methods struggle to adapt quickly to the demands of diverse nursing procedures, limiting their flexible application in teaching.

[0003] In order to overcome the above problems, in recent years, technologies based on computer vision and deep learning have been introduced into the field of VR content generation. The development of large multimodal models (such as vision-language joint models) has made it possible to extract 3D information from real videos and automatically generate virtual scenes. Through video analysis, motion capture and 3D reconstruction technology, real nursing operation videos can be efficiently converted into interactive VR teaching models. For example, using human posture estimation and scene understanding technology, key actions in the video can be automatically identified and replaced with 3D animation elements, while the interactive experience is enhanced through tactile feedback embedding technology. Therefore, there is an urgent need for a method that can quickly generate interactive pediatric nursing VR models based on real training videos to improve teaching efficiency, reduce development costs, and enhance students' practical operation capabilities. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention proposes a method and system for generating a pediatric nursing practice teaching VR model based on training videos. Through video analysis, 3D element replacement and animation embedding technology, the intelligent generation of pediatric nursing VR teaching resources is realized, providing a more efficient and immersive solution for nursing education.

[0005] The first aspect of the present invention provides a method for generating a VR model for pediatric nursing practice teaching based on a training video, comprising the following steps:

[0006] Collecting pediatric nursing operation videos, preprocessing the videos, extracting key action frames, and separating medical staff, patients, and medical tools;

[0007] Convert medical staff, patients, and medical tools in pediatric nursing operation videos into editable 3D models, obtain multimodal data corresponding to pediatric nursing operations, and use the multimodal large model to match the converted 3D actions with semantics to optimize 3D conversion;

[0008] Mark key contact points in the 3D action sequence, perform collision detection and tracking at the key contact points, and define the operation process to generate a 3D animation sequence with interactive logic;

[0009] A VR teaching scene for pediatric nursing training was constructed based on the 3D model and 3D animation sequence with interactive logic. The rationality of the operation logic was verified through user testing, and the scene was modified using the verification results.

[0010] In this solution, pediatric nursing operation videos are collected and pre-processed to extract key action frames and separate medical staff, children, and medical tools. Specifically:

[0011] Using a synchronized multi-camera system to record standardized pediatric nursing procedures, obtaining the procedure flow based on the category labels of the pediatric nursing procedures, and automatically segmenting the video using a dynamic time warping algorithm based on the procedure flow;

[0012] Calculate inter-frame motion energy and HSV histogram differences to extract candidate key frames, extract image features and temporal features of the candidate key frames, select key action frames in each video segment based on the image features and temporal features, and mark the timestamps of the key action frames;

[0013] The key action frames of each video segment are imported into the improved YOLOv8 model. SPD-Conv is used in the YOLOv8 model to replace the downsampling. The feature maps of the input key action frames are split according to the spatial dimension, merged into the channel dimension, and then the channel is compressed through one-dimensional convolution.

[0014] A multi-scale attention module is inserted after SPD-Conv. It captures features of different scales through parallel dilated convolutions, and then weights them through channel attention. A bidirectional cross-scale connection is set in the FPN network to perform bidirectional fusion of the weighted features of different scales. Gated convolution is used to dynamically adjust the feature weights of each layer to obtain aggregated features and import them into the detection head.

[0015] The spatial, channel, and scale attention of the detection head are dynamically adjusted through learnable parameters, deformable convolution is introduced into the detection head, and convolution learning is performed on the aggregated features. Medical staff, children, and medical tools in key action frames are identified through classification detection and regression detection, and the separated binary masks of medical staff, children, and medical tools are output.

[0016] In this solution, the medical staff, patients, and medical tools in the pediatric nursing operation video are converted into editable 3D models. Specifically:

[0017] Performing posture estimation on medical staff and children, obtaining 3D posture sequences of medical staff and children, mapping the 3D posture sequences to 3D virtual characters, performing body shape adaptation on the 3D virtual characters, and establishing skeleton mapping rules for posture reconstruction;

[0018] Extracting geometric features from the medical tool mask, searching the geometric features in a preset medical tool 3D asset library using similarity calculation to obtain a 3D model corresponding to the medical tool;

[0019] Multi-source point cloud data of the ward environment is collected, and layered reconstruction of the ward environment is performed based on the multi-source point cloud data to construct a 3D ward environment.

[0020] In this solution, the postures of medical staff and patients are estimated to obtain their 3D posture sequences, specifically:

[0021] Obtain video segments constructed with masks of medical staff, children, and medical tools, extract features from the spatial and temporal dimensions, and construct action branches based on temporal modeling. Input continuous frames are passed through a 3D CNN network to extract temporal features, which are then encoded using a temporal Transformer encoder. Asymmetric temporal sampling is used to obtain inter-frame temporal dependencies, and the action category features of medical staff and children are output.

[0022] The pose branch is constructed by modeling the spatial dimension, inputting the intermediate frame, extracting spatial features through the 2D CNN network, and regressing the joint point coordinates through the fully connected layer based on the spatial features to generate pose features;

[0023] The action category features and posture features are respectively subjected to attention encoding, an attention matrix is ​​generated as an action-posture correlation matrix through an attention mechanism, the posture features are weighted using the action-posture correlation matrix, and the weighted posture features are added to the original posture features to obtain a fusion feature;

[0024] An initial 3D pose estimate is generated based on the pose features. The initial 3D pose estimate and the fused features are input into a graph convolutional network. The action category features are used to project them into joint motion constraint vectors. The joint motion constraint vectors are used to correct the pose in each layer update of the graph convolutional network to obtain the final pose information of the medical staff and the child.

[0025] In this solution, we acquire multimodal data corresponding to pediatric nursing operations, match the converted 3D actions with semantics through a multimodal large model, and optimize the 3D conversion. Specifically,

[0026] Obtaining video frames, text instructions, and voice explanations corresponding to pediatric nursing procedures to generate multimodal data, aligning and preprocessing the multimodal data, and extracting frame-level features, step features, and intonation features through the visual encoder, text encoder, and voice encoder of the multimodal large model;

[0027] Multimodal fusion is performed through hierarchical attention. The cross-attention mechanism is used to establish the association between vision and text, output enhanced visual features, and obtain the association between vision and speech through the gating mechanism. When warning words appear in the speech, the features of the relevant area are enhanced to achieve speech correction and finally obtain semantic features.

[0028] Using contrastive learning to constrain the semantic space, a multi-layer perceptron is used to calculate semantically relevant joint weights for the 3D posture sequences obtained by the posture estimation of medical staff and children based on the semantic features. Semantic-guided attention is introduced for enhancement to obtain joint point weights, and key point positions are optimized and adjusted based on the joint point weights.

[0029] The integrity of the process is monitored through the LSTM hidden state. The 3D pose sequence and semantic features are imported into the LSTM network to obtain the hidden state. The state transition matrix of the child care operation is defined. The current state probability is predicted using Softmax. The state anomaly is detected by comparing the current state probability with the state transition matrix.

[0030] When the state is abnormal, global reoptimization is triggered and inverse kinematics chain correction is used.

[0031] In this solution, key contact points are marked in the 3D action sequence, collision detection is performed at these key contact points, and the operation process is defined to generate a 3D animation sequence with interactive logic. Specifically:

[0032] Acquire corresponding multimodal data in the 3D action sequence, use the multimodal data to perform multimodal action recognition, mark key contact points, perform three-dimensional calibration on the key contact points, and establish a medical tool collision body hierarchy in Blender;

[0033] The finite state machine is used to construct operation process state nodes and transfer conditions to generate pediatric nursing operation processes. Tactile feedback and visual feedback are set in the operation process state nodes to generate 3D animation sequences with interactive logic.

[0034] In this solution, a VR teaching scene for pediatric nursing training was constructed based on the 3D model and 3D animation sequence with interactive logic. The rationality of the operation logic was verified through user testing, and the scene was modified using the verification results. Specifically:

[0035] Import the reconstructed 3D ward environment, deploy 3D models of medical staff, patients, and medical tools in the 3D ward environment according to the functional zoning layout, implement the interactive logic within the scene based on the 3D animation sequence with interactive logic, and build a VR teaching scene for pediatric nursing training;

[0036] Collect the operation trajectory and event logs during the user test, analyze the operation trajectory, generate a hand movement heat map and hand operation key frame comparison results, and perform verification and evaluation based on the event log according to the preset evaluation dimensions;

[0037] Based on the verification and evaluation results, the problem types in the user testing process are identified, and solutions are retrieved based on the problem types using big data methods to perform logical optimization and model updates on the VR teaching scenarios for pediatric nursing training.

[0038] The second aspect of the present invention provides a pediatric nursing practice teaching VR model generation system based on training videos, including: a video acquisition and preprocessing module, a multimodal 3D conversion module, an interactive animation tracking module, a VR model generation and rendering module, and a teaching application and feedback optimization module;

[0039] The video acquisition and preprocessing module acquires and preprocesses pediatric nursing operation videos, extracts key action frames, separates medical staff, patients, and medical tools, and removes irrelevant background.

[0040] The multimodal 3D conversion module converts the medical staff, children, and medical tools in the pediatric nursing operation video into editable 3D models, obtains the multimodal data corresponding to the pediatric nursing operation, and matches the converted 3D actions with semantics through the multimodal large model to optimize the 3D conversion;

[0041] The interactive animation tracking module marks key contact points in the 3D action sequence, performs collision detection tracking on the key contact points, and defines the operation process to generate a 3D animation sequence with interactive logic to achieve tactile feedback and operation guidance;

[0042] The VR model generation and rendering module constructs a VR teaching scene for pediatric nursing training based on the 3D model and the 3D animation sequence with interactive logic;

[0043] The teaching application and feedback optimization module verifies the rationality of the operation logic through user testing, uses the verification results to correct the scenario, and generates a learning analysis report for the user.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] This invention solves the core problems of low efficiency, high cost, and lack of authenticity in traditional pediatric nursing training through the automated generation process of "video → 3D → VR", while enabling personalized teaching through intelligent interaction and data analysis. 3D models are generated directly from videos based on multimodal large models, reducing manual intervention, and can quickly process a large number of training videos to adapt to different nursing operations. Physical collision detection is implanted at key contact points to trigger real-time feedback. Students can efficiently master skills through the "try-feedback-correction" cycle, reduce error rates, and enhance the interactive learning experience. High-risk operations can be repeated an unlimited number of times in a VR environment, reducing training costs and risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments or exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or exemplary descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained according to these drawings without paying any creative work.

[0047] Figure 1 The flowchart of the method for generating a VR model for pediatric nursing practice teaching based on training videos is shown;

[0048] Figure 2 A flowchart of posture estimation for medical staff and children is shown;

[0049] Figure 3 A flow chart for optimizing 3D transformation via a multimodal large model is shown;

[0050] Figure 4 The block diagram of the pediatric nursing practice teaching VR model generation system based on training videos is shown. DETAILED DESCRIPTION

[0051] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features therein can be combined with each other.

[0052] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0053] Figure 1 A flowchart of the method for generating a VR model for pediatric nursing practice teaching based on training videos is shown.

[0054] like Figure 1As shown, this embodiment provides a method for generating a pediatric nursing practice teaching VR model based on a training video, including:

[0055] S102, collecting pediatric nursing operation videos, pre-processing the pediatric nursing operation videos, extracting key action frames, and separating medical staff, children, and medical tools;

[0056] S104: Convert the medical staff, patients, and medical tools in the pediatric nursing operation video into editable 3D models, obtain multimodal data corresponding to the pediatric nursing operation, match the converted 3D actions with semantics through the multimodal large model, and optimize the 3D conversion;

[0057] S106, marking key contact points in the 3D action sequence, performing collision detection and tracking at the key contact points, and defining an operation process to generate a 3D animation sequence with interactive logic;

[0058] S108, constructing a VR teaching scene for pediatric nursing training based on the 3D model and the 3D animation sequence with interactive logic, verifying the rationality of the operation logic through user testing, and using the verification results to modify the scene.

[0059] It should be noted that a synchronized multi-camera system was used to record standardized pediatric nursing operations, such as intravenous puncture, medication feeding, and cardiopulmonary resuscitation, to ensure comprehensive coverage of the actions. The operation process was obtained based on the category labels of the pediatric nursing operations. Based on the operation process, a dynamic time warping algorithm was used to automatically segment the video, for example: disinfection (0-5s) - puncture (5-10s) - fixation (10-15s). The inter-frame motion energy was calculated to select frames with sudden motion changes. The HSV histogram difference was used to select frames with obvious content changes, and candidate key frames were extracted. The frame-level feature vectors of the candidate key frames were extracted as image features based on the pre-trained CNN. The continuous frames were encoded using Bi-LSTM, and the turning points of the action were marked to obtain temporal features. The key action frames of each video segment were selected based on the image features and temporal features, and the timestamps of the key action frames were marked.

[0060] The YOLOv8 model is improved to solve the problem of insufficient detection accuracy of small targets such as needles and stethoscopes in traditional models. The key action frames of each video segment are imported into the improved YOLOv8 model. SPD-Conv is used in the YOLOv8 model to replace downsampling. The feature maps of the input key action frames are split according to the spatial dimension and merged into the channel dimension. The channel is then compressed through one-dimensional convolution. SPD-Conv retains all pixel information to avoid information loss in downsampling. A multi-scale attention module is inserted after SPD-Conv. Different scale features are captured through parallel hole convolutions, and then weighted through channel attention to enhance sensitivity to small targets (such as needle tips and cotton balls). Since traditional FPN suffers from information loss when fusing multiple layers of features, bidirectional cross-scale connections are set in the FPN network to perform bidirectional fusion of weighted different scale features, transferring semantic information from top to bottom and fine-grained information from bottom to top. Gate convolution is used to dynamically adjust the feature weights of each layer, which is expressed as:

[0061] F out =σ(W g ·[F high ,F low ])⊙F high +(1-σ)⊙F low

[0062] Among them F out represents the aggregated features, σ represents the Sigmoid activation function, W g represents the gating weight matrix, F high represents high-level features, F low represents low-level features, ⊙ represents element-wise multiplication;

[0063] Aggregated features are captured and imported into the detection head. Learnable parameters are used to dynamically adjust the detection head's spatial, channel, and scale attention. Deformable convolution is introduced into the detection head to adapt to irregular targets. Convolutional learning is performed on the aggregated features. Classification and regression detection are used to identify medical staff, children, and medical tools in key action frames. Binary masks of the separated medical staff, children, and medical tools are output. The GrabCut algorithm is applied to the output masks to refine edges, such as the contact area between a child's finger and a syringe. In training the improved YOLOv8 model, labeled pediatric nursing video frames are used as training data, and the backbone network and the detection head are trained alternately.

[0064] It should be noted that the postures of medical staff and children are estimated to obtain their 3D posture sequences. The basic body templates of medical staff and children are loaded according to the role type, and the 3D posture sequence is mapped to the 3D virtual character. The 3D virtual character is adapted to the body shape through features such as bone length ratio constraints, and a skeleton mapping rule is established for posture reconstruction. Inverse kinematic constraints are added to special actions (such as kneeling operations). For actions such as crying children, expression bases and skeletal animations are mixed. Geometric features such as contour Hu moment and principal axis direction are extracted according to the mask of the medical tool, and the geometric features are retrieved in the preset medical tool 3D asset library using similarity calculation to obtain the 3D model corresponding to the medical tool; multi-source point cloud data of the ward environment is collected according to the scanning path by handheld RGB-D scanners and other equipment, and the geometric reconstruction, semantic enhancement and functional area division of the ward environment are performed according to the multi-source point cloud data to construct a 3D ward environment and inject dynamic elements into interactive objects.

[0065] Figure 2 A flowchart of posture estimation for medical staff and children is shown.

[0066] According to an embodiment of the present invention, the posture of the medical staff and the patient is estimated to obtain a 3D posture sequence of the medical staff and the patient, specifically:

[0067] S202: Obtain video segments constructed with masks of medical staff, children, and medical tools, extract features from the spatial and temporal dimensions, model and construct action branches from the temporal dimension, input continuous frames through a 3D CNN network to extract temporal features, and encode the temporal features using a temporal Transformer encoder. Asymmetric temporal sampling is used to obtain inter-frame temporal dependencies, and action category features of medical staff and children are output.

[0068] S204, constructing a posture branch from spatial dimension modeling, inputting the intermediate frame, extracting spatial features through the 2D CNN network, and regressing the joint point coordinates through the fully connected layer based on the spatial features to generate posture features;

[0069] S206, performing attention encoding on the action category features and the posture features respectively, generating an attention matrix as an action-posture correlation matrix through an attention mechanism, weighting the posture features using the action-posture correlation matrix, and adding the weighted posture features to the original posture features to obtain a fusion feature;

[0070] S208, generate an initial 3D pose estimate based on the pose features, input the initial 3D pose estimate and the fusion features into the graph convolutional network, use the action category features to project them into joint motion constraint vectors, use the joint motion constraint vectors to correct the pose in each layer update of the graph convolutional network, and obtain the final pose information of the medical staff and the child.

[0071] It should be noted that features are extracted from the spatial dimension (single-frame pose) and the temporal dimension (continuous frame action) respectively through a parallel dual-branch network, and high-precision estimation of three-dimensional pose is achieved through cross-branch attention fusion. The action branch learns the temporal pattern of the action, inputs continuous T frames, extracts temporal features through 3D CNN (such as SlowFast), uses the temporal Transformer encoder to model inter-frame dependencies, outputs action category features, and adopts asymmetric temporal sampling in action classification, sampling key action frames at high frequency (such as before and after the needle touches the skin), and sampling static frames at low frequency. In the posture branch, the 3D human joints of a single frame are estimated, the target frame is input, the spatial features are extracted through 2D CNN (such as HRNet), the joint coordinates are regressed through the fully connected layer, and the initial pose estimate is generated. The action category features are projected into the query vector Q, the spatial features are flattened into the key vector K and the value vector V, and the action-posture correlation matrix is ​​calculated. d is the dimension, and the posture features are weighted F=A·V so that the action semantics affect the positioning of the joint points. For example, the wrist needs to be kept stable for puncture. The weighted posture features are added to the original posture features through residual connections to obtain fusion features. The human joint tree structure is used to convert the action category semantics into motion range constraints, and a graph structure is constructed. Each human node feature is the initial posture coordinate and the corresponding part of the fusion feature. An edge structure is established based on the human skeleton connection relationship, and the action category features are projected into joint motion constraint vectors. For example: if the action is puncture, the joint motion constraint vector of the wrist joint is to limit the rotation range. The posture is gradually refined through 3 layers of GCN, and the corrected posture is output. Preferably, LSTM is used to smooth the postures of the previous and next frames to avoid jitter.

[0072] Figure 3 A flow chart for optimizing 3D transformation via a multimodal large model is shown.

[0073] According to an embodiment of the present invention, multimodal data corresponding to pediatric nursing operations is obtained, and the converted 3D actions are matched with semantics through a multimodal large model to optimize the 3D conversion, specifically:

[0074] S302: Obtain video frames, text operation instructions, and voice explanation data corresponding to pediatric nursing operations to generate multimodal data, align and preprocess the multimodal data, and extract frame-level features, step features, and intonation features using the visual encoder, text encoder, and voice encoder of the multimodal large model;

[0075] S304: Multimodal fusion is performed through hierarchical attention. A cross-attention mechanism is used to establish the association between vision and text, output enhanced visual features, and obtain the association between vision and speech through a gating mechanism. When a warning word appears in the speech, the features of the relevant area are enhanced to achieve speech correction and finally obtain semantic features.

[0076] S306, using contrastive learning to constrain the semantic space, calculating semantically relevant joint weights for the 3D posture sequence obtained by the posture estimation of the medical staff and the child using a multi-layer perceptron based on the semantic features, introducing semantically guided attention for enhancement, obtaining joint point weights, and optimizing and adjusting the key point positions based on the joint point weights;

[0077] S308, supervising the process integrity through the LSTM hidden state, importing the 3D posture sequence and semantic features into the LSTM network to obtain the hidden state, defining the state transition matrix of the child care operation, using Softmax to predict the current state probability, and obtaining the state abnormality based on the comparison between the current state probability and the state transition matrix;

[0078] S310, when the state is abnormal, triggers global re-optimization and uses inverse kinematics chain correction.

[0079] It should be noted that the CL4CT model maps the three types of data corresponding to pediatric nursing operations, namely, video frames, text operation instructions, and voice explanation data, into a unified 128-dimensional space. The visual encoder uses CLIP-ViT to extract frame-level features F. v , select the operation turning point; the text encoder uses ClinicalBERT to generate step features F t , converted into a structural step; the speech encoder uses Wav2Vec2.0 to extract the intonation feature F a , extract emphasis words, such as special attention to aseptic operation; establish visual-text association through cross-attention mechanism W q 、W k is a learnable parameter that outputs enhanced visual features F v ′ =A vt F t , construct the speech-activated visual feature gate g = σ(W g [F v ′ ; F a ]), σ is the activation function, and the gating mechanism is used to fuse the semantic features F final =g⊙F v ′ +(1-g)⊙F a .

[0080] Joint importance is predicted through MLP. Semantic features are input to concatenate joint coordinates. Joint weights are obtained through three fully connected layers and activation functions. For example, in venipuncture, high weights are assigned to the wrist and metacarpophalangeal joints based on needle insertion angle and stability. Joint positions are then adjusted based on the joint weights. Indicates the position of the joint point after adjustment, w j ′ is the joint weight, Δp j The semantic features regressed by the MLP are used. LSTM is used for process supervision and posture correction. The hidden state is obtained through LSTM state detection, and the current state probability is predicted through the classification head. State anomalies are identified based on the state transition matrix of predefined pediatric nursing operations. For example, unsterilized puncture triggers an inverse kinematic chain correction, correcting the wrist position, elbow angle, and shoulder posture.

[0081] It should be noted that corresponding multimodal data is acquired from 3D action sequences and used for multimodal action recognition. For example, fine movements are identified by capturing posture changes, action segments are analyzed, step classifications are output, and key instructions in the voice explanation are simultaneously verified. Key contact points are marked and three-dimensionally calibrated. A medical tool collision hierarchy is established in Blender, for example, a 2mm radius sphere corresponds to a tweezer, and a 5mm capsule corresponds to a syringe. A finite state machine is used to construct operation process state nodes and transition conditions. For example, state nodes include preparation, disinfection, puncture, and completion. The transition condition from the preparation node to the disinfection node is that the hand is wearing gloves, the transition condition from the disinfection node to the puncture node is that the cotton ball has been disinfected more than three times, and the transition condition from the puncture node to the completion node is that blood returns and is fixed. A pediatric nursing operation process is generated, and tactile and visual feedback are set in the operation process state nodes to generate a 3D animation sequence with interactive logic. For example, force feedback of different intensities is generated when the cotton ball touches the skin, the needle enters the blood vessel, and the tape is applied. The cotton ball illuminates during disinfection, and the blood vessel is displayed when the puncture is successful. When an illegal operation is detected (such as direct puncture without disinfection), the VR scene flashes red and is forced to reset to the last correct state, playing a standard operation demonstration animation. Through precise spatiotemporal event mapping and state control that conforms to clinical logic, a highly immersive VR training experience is provided while ensuring medical standardization. The system specifically optimizes high-frequency contact events in pediatric nursing (such as dynamic collision detection when a child's limbs move), making virtual training as close to real clinical scenarios as possible.

[0082] It should be noted that the reconstructed 3D ward environment was imported, arranged according to functional zoning, and dynamic lighting was set up. 3D models of medical staff, children, and medical tools were deployed in the 3D ward environment. IK inverse kinematics adjustment was supported, and the interactive logic within the scene was implemented based on 3D animation sequences with interactive logic, thus constructing a VR teaching scene for pediatric nursing training.

[0083] During user testing, operation traces and event logs were collected. These event logs record state transition timestamps and trigger conditions. These traces were analyzed to generate hand movement heatmaps and keyframe comparisons of hand movements with standard procedures. Based on these event logs, a verification assessment was conducted according to pre-defined evaluation criteria, such as step completeness, angle and force accuracy, feedback timeliness, and scene realism. Based on these verification and assessment results, problem types during user testing were identified. Solutions were retrieved using big data methods. The logic of the VR teaching scene for pediatric nursing training was optimized and the model was updated. Based on the solutions, state machine condition checks were expanded and fault tolerance mechanisms were added. For example, the original condition was a cotton ball contacting the skin, while the revised condition was a contact area >50% and duration >2 seconds. The VR teaching scene for pediatric nursing training was updated by expanding the patient response library and the physical properties of medical tools. If an incorrect operation occurred during user testing and verification, such as puncturing without complete disinfection, the VR teaching scene would immediately freeze and play a standard operation video. An interactive quiz was inserted at the error point: "Select the correct disinfection range."

[0084] In the actual operation of the VR teaching scenario of pediatric nursing training, users practice nursing operations through headsets and handles. The system scores in real time, analyzes user operation data, provides voice and text guidance, collects user feedback, and generates learning analysis reports to adjust the 3D model accuracy or teaching difficulty.

[0085] Figure 4 The block diagram of the pediatric nursing practice teaching VR model generation system based on training videos is shown.

[0086] The second embodiment of the present invention provides a pediatric nursing practice teaching VR model generation system 4 based on training videos, including: a video acquisition and preprocessing module 401, a multimodal 3D conversion module 402, an interactive animation tracking module 403, a VR model generation and rendering module 404, and a teaching application and feedback optimization module 405;

[0087] The video acquisition and preprocessing module acquires and preprocesses pediatric nursing operation videos, extracts key action frames, separates medical staff, patients, and medical tools, and removes irrelevant background.

[0088] The multimodal 3D conversion module converts the medical staff, children, and medical tools in the pediatric nursing operation video into editable 3D models, obtains the multimodal data corresponding to the pediatric nursing operation, and matches the converted 3D actions with semantics through the multimodal large model to optimize the 3D conversion;

[0089] The interactive animation tracking module marks key contact points in the 3D action sequence, performs collision detection tracking on the key contact points, and defines the operation process to generate a 3D animation sequence with interactive logic to achieve tactile feedback and operation guidance;

[0090] The VR model generation and rendering module constructs a VR teaching scene for pediatric nursing training based on the 3D model and the 3D animation sequence with interactive logic;

[0091] The teaching application and feedback optimization module verifies the rationality of the operation logic through user testing, uses the verification results to correct the scenario, and generates a learning analysis report for the user.

[0092] The third aspect of the present invention provides a computer-readable storage medium, which includes a pediatric nursing practice teaching VR model generation method program based on a training video. When the pediatric nursing practice teaching VR model generation method program based on a training video is executed by a processor, it implements the steps of the pediatric nursing practice teaching VR model generation method based on a training video.

[0093] In the several embodiments provided in this application, it should be understood that the disclosed methods and systems can be implemented in other ways. The system embodiments described above are merely illustrative. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms. In addition, the functional modules in the various embodiments of the present invention can all be integrated into one processing module, or each module can be a separate module, or two or more modules can be integrated into one module; the above-mentioned integrated modules can be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0094] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disks or optical disks, and other media that can store program codes.

[0095] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be covered by the scope of protection of the present invention.

Claims

1. A method for generating a VR model for pediatric nursing practice teaching based on training videos, characterized in that: The following steps are involved: Collecting pediatric nursing operation videos, preprocessing the videos, extracting key action frames, and separating medical staff, patients, and medical tools; Convert medical staff, patients, and medical tools in pediatric nursing operation videos into editable 3D models, obtain multimodal data corresponding to pediatric nursing operations, and use the multimodal large model to match the converted 3D actions with semantics to optimize 3D conversion; Mark key contact points in the 3D action sequence, perform collision detection and tracking at the key contact points, and define the operation process to generate a 3D animation sequence with interactive logic; A VR teaching scene for pediatric nursing training was constructed based on the 3D model and 3D animation sequence with interactive logic. The rationality of the operation logic was verified through user testing, and the scene was modified using the verification results.

2. The method for generating a VR model for pediatric nursing practice teaching based on training videos according to claim 1 is characterized in that: Collect pediatric nursing operation videos, pre-process the videos, extract key action frames, and separate medical staff, children, and medical tools. Specifically: Using a synchronized multi-camera system to record standardized pediatric nursing procedures, obtaining the procedure flow based on the category labels of the pediatric nursing procedures, and automatically segmenting the video using a dynamic time warping algorithm based on the procedure flow; Calculate inter-frame motion energy and HSV histogram differences to extract candidate key frames, extract image features and temporal features of the candidate key frames, select key action frames in each video segment based on the image features and temporal features, and mark the timestamps of the key action frames; The key action frames of each video segment are imported into the improved YOLOv8 model. SPD-Conv is used in the YOLOv8 model to replace the downsampling. The feature maps of the input key action frames are split according to the spatial dimension, merged into the channel dimension, and then the channel is compressed through one-dimensional convolution. A multi-scale attention module is inserted after SPD-Conv. It captures features of different scales through parallel dilated convolutions, and then weights them through channel attention. A bidirectional cross-scale connection is set in the FPN network to perform bidirectional fusion of the weighted features of different scales. Gated convolution is used to dynamically adjust the feature weights of each layer to obtain aggregated features and import them into the detection head. The spatial, channel, and scale attention of the detection head are dynamically adjusted through learnable parameters, deformable convolution is introduced into the detection head, and convolution learning is performed on the aggregated features. Medical staff, children, and medical tools in key action frames are identified through classification detection and regression detection, and the separated binary masks of medical staff, children, and medical tools are output.

3. The method for generating a VR model for pediatric nursing practice teaching based on training videos according to claim 1 is characterized in that: Convert medical staff, patients, and medical tools in pediatric nursing operation videos into editable 3D models, specifically: Performing posture estimation on medical staff and children, obtaining 3D posture sequences of medical staff and children, mapping the 3D posture sequences to 3D virtual characters, performing body shape adaptation on the 3D virtual characters, and establishing skeleton mapping rules for posture reconstruction; Extracting geometric features from the medical tool mask, searching the geometric features in a preset medical tool 3D asset library using similarity calculation to obtain a 3D model corresponding to the medical tool; Multi-source point cloud data of the ward environment is collected, and layered reconstruction of the ward environment is performed based on the multi-source point cloud data to construct a 3D ward environment.

4. The method for generating a VR model for pediatric nursing practice teaching based on training videos according to claim 3 is characterized in that: Perform posture estimation on the medical staff and the patient to obtain their 3D posture sequences, specifically: Obtain video segments constructed with masks of medical staff, children, and medical tools, extract features from the spatial and temporal dimensions, and construct action branches based on temporal modeling. Input continuous frames are passed through a 3D CNN network to extract temporal features, which are then encoded using a temporal Transformer encoder. Asymmetric temporal sampling is used to obtain inter-frame temporal dependencies, and the action category features of medical staff and children are output. The pose branch is constructed by modeling the spatial dimension, inputting the intermediate frame, extracting spatial features through the 2D CNN network, and regressing the joint point coordinates through the fully connected layer based on the spatial features to generate pose features; The action category features and posture features are respectively subjected to attention encoding, an attention matrix is ​​generated as an action-posture correlation matrix through an attention mechanism, the posture features are weighted using the action-posture correlation matrix, and the weighted posture features are added to the original posture features to obtain a fusion feature; An initial 3D pose estimate is generated based on the pose features. The initial 3D pose estimate and the fused features are input into a graph convolutional network. The action category features are used to project them into joint motion constraint vectors. The joint motion constraint vectors are used to correct the pose in each layer update of the graph convolutional network to obtain the final pose information of the medical staff and the child.

5. The method for generating a VR model for pediatric nursing practice teaching based on training videos according to claim 3 is characterized in that: Obtain multimodal data corresponding to pediatric nursing operations, match the converted 3D actions with semantics through a multimodal large model, and optimize the 3D conversion. Specifically: Obtaining video frames, text instructions, and voice explanations corresponding to pediatric nursing procedures to generate multimodal data, aligning and preprocessing the multimodal data, and extracting frame-level features, step features, and intonation features through the visual encoder, text encoder, and voice encoder of the multimodal large model; Multimodal fusion is performed through hierarchical attention. The cross-attention mechanism is used to establish the association between vision and text, output enhanced visual features, and obtain the association between vision and speech through the gating mechanism. When warning words appear in the speech, the features of the relevant area are enhanced to achieve speech correction and finally obtain semantic features. Using contrastive learning to constrain the semantic space, a multi-layer perceptron is used to calculate semantically relevant joint weights for the 3D posture sequences obtained by the posture estimation of medical staff and children based on the semantic features. Semantic-guided attention is introduced for enhancement to obtain joint point weights, and key point positions are optimized and adjusted based on the joint point weights. The integrity of the process is monitored through the LSTM hidden state. The 3D pose sequence and semantic features are imported into the LSTM network to obtain the hidden state. The state transition matrix of the child care operation is defined. The current state probability is predicted using Softmax. The state anomaly is detected by comparing the current state probability with the state transition matrix. When the state is abnormal, global reoptimization is triggered and inverse kinematics chain correction is used.

6. The method for generating a VR model for pediatric nursing practice teaching based on training videos according to claim 1 is characterized in that: Mark key contact points in the 3D action sequence, perform collision detection and tracking at these key contact points, and define the operation process to generate a 3D animation sequence with interactive logic. Specifically: Acquire corresponding multimodal data in the 3D action sequence, use the multimodal data to perform multimodal action recognition, mark key contact points, perform three-dimensional calibration on the key contact points, and establish a medical tool collision body hierarchy in Blender; The finite state machine is used to construct operation process state nodes and transfer conditions to generate pediatric nursing operation processes. Tactile feedback and visual feedback are set in the operation process state nodes to generate 3D animation sequences with interactive logic.

7. The method for generating a VR model for pediatric nursing practice teaching based on training videos according to claim 1 is characterized in that: Based on the 3D model and 3D animation sequence with interactive logic, a VR teaching scene for pediatric nursing training was constructed. The rationality of the operation logic was verified through user testing, and the scene was modified using the verification results. Specifically: Import the reconstructed 3D ward environment, deploy 3D models of medical staff, patients, and medical tools in the 3D ward environment according to the functional zoning layout, implement the interactive logic within the scene based on the 3D animation sequence with interactive logic, and build a VR teaching scene for pediatric nursing training; Collect the operation trajectory and event logs during the user test, analyze the operation trajectory, generate a hand movement heat map and hand operation key frame comparison results, and perform verification and evaluation based on the event log according to the preset evaluation dimensions; Based on the verification and evaluation results, the problem types in the user testing process are identified, and solutions are retrieved based on the problem types using big data methods to perform logical optimization and model updates on the VR teaching scenarios for pediatric nursing training.

8. A pediatric nursing practice teaching VR model generation system based on training videos, characterized by: A method for generating a VR model for practical teaching of pediatric nursing based on training videos as described in any one of claims 1 to 7, comprising: a video acquisition and preprocessing module, a multimodal 3D conversion module, an interactive animation tracking module, a VR model generation and rendering module, and a teaching application and feedback optimization module; The video acquisition and preprocessing module acquires and preprocesses pediatric nursing operation videos, extracts key action frames, separates medical staff, patients, and medical tools, and removes irrelevant background. The multimodal 3D conversion module converts the medical staff, children, and medical tools in the pediatric nursing operation video into editable 3D models, obtains the multimodal data corresponding to the pediatric nursing operation, and matches the converted 3D actions with semantics through the multimodal large model to optimize the 3D conversion; The interactive animation tracking module marks key contact points in the 3D action sequence, performs collision detection tracking on the key contact points, and defines the operation process to generate a 3D animation sequence with interactive logic to achieve tactile feedback and operation guidance; The VR model generation and rendering module constructs a VR teaching scene for pediatric nursing training based on the 3D model and the 3D animation sequence with interactive logic; The teaching application and feedback optimization module verifies the rationality of the operation logic through user testing, uses the verification results to correct the scenario, and generates a learning analysis report for the user.

Images