Interventional nursing teaching simulation system based on multi-modal interaction

Abstract

The invention belongs to the field of teaching simulation, and particularly discloses an interventional nursing teaching simulation system based on multi-modal interaction, which comprises an organization simulation module, a multi-modal interaction module and an operation track analysis module. According to the scheme, through a biomechanical simulation method based on position dynamics and a multi-constraint mechanism, superficial fascia is subjected to two-stage solver, distance keeping constraint, dihedral angle constraint and linear and angular damping force modeling, and IMA blood vessels are subjected to combination of high-rigidity distance constraint and low-rigidity dihedral angle constraint and a high-low resolution grid balance strategy; the normal resistance and the tangential resistance are used for constructing multi-dimensional tactile feedback force, and accurate reduction of mechanical properties of different human tissues is achieved; through a deep learning algorithm, hyper-parameters are optimized through grid search, evaluation indexes are calculated through a weighted average method, the significance is verified by comparing frame number differences of simulation operation and standard operation in all stages and combining independent sample t verification, and a complete quantification system of operation track evaluation is constructed.

Description

Technical Field

[0001] This invention relates to the field of teaching simulation, specifically to an interventional nursing teaching simulation system based on multimodal interaction. Background Technology

[0002] Multimodal interactive interventional nursing teaching simulation is an innovative teaching model that integrates multiple interactive methods and technologies to simulate real clinical scenarios for teaching activities, improving clinical practice skills and reducing teaching costs and risks. Existing teaching simulation systems suffer from low fidelity in replicating the biomechanical properties of human tissues, inaccurate surgical operation simulations, and a lack of realistic tactile feedback. This results in a lack of clinical realism in surgical operation simulations, hindering the effective accumulation of muscle memory and judgment experience required for clinical practice, and significantly impacting the cultivation of practical skills. Furthermore, existing simulation systems suffer from a disconnect between multimodal interaction and operational assessment, insufficient teaching focus, lack of dynamic adaptability, and difficulty in meeting the learning needs of different learners, leading to low teaching efficiency and an inability to achieve precise empowerment. Summary of the Invention

[0003] To address the above issues and overcome the shortcomings of existing technologies, this invention provides a multimodal interactive interventional nursing teaching simulation system. Addressing the technical problems of low fidelity in replicating the biomechanical properties of human tissues, inaccurate surgical operation simulation, and lack of realistic tactile feedback in existing teaching simulation systems—resulting in a lack of clinical realism and an inability to effectively accumulate the muscle memory and judgment experience required for clinical practice, thus significantly impacting the cultivation of practical skills—this solution utilizes a biomechanical simulation method based on positional dynamics and multi-constraint mechanisms. For the superficial fascia, a two-stage solver, distance-preserving constraints, dihedral constraints, and linear and angular damping forces are used for modeling. For the IMA vessels, a combination of high-stiffness distance constraints and low-stiffness dihedral constraints, along with a high- and low-resolution mesh balancing strategy, is employed. For surgical operations, topology-preserving cutting algorithms and connective tissue cutting algorithms are used. This solution simulates core movements and constructs multi-dimensional tactile feedback forces using normal and tangential resistance to accurately reproduce the biomechanical properties of different human tissues, providing a realistic tactile interaction experience close to clinical practice. This effectively improves trainees' operational proficiency, movement standardization, and emergency response capabilities. Existing simulation systems suffer from technical problems such as a disconnect between multimodal interaction and operational assessment, insufficient teaching focus, lack of dynamic adaptability, and difficulty in meeting the learning needs of different trainees, resulting in low teaching efficiency and an inability to provide precise empowerment. This solution utilizes deep learning algorithms, employs grid search to optimize hyperparameters, and calculates assessment indicators using a weighted average method. By comparing the frame count differences between simulated and standard operations at each stage, and verifying significance using independent samples t-tests, a complete quantitative system for operational trajectory assessment is constructed, enabling accurate identification of surgical stages and improving the relevance and efficiency of teaching.

[0004] The technical solution adopted by the present invention is as follows: The present invention provides an interventional nursing teaching simulation system based on multimodal interaction, the interventional nursing teaching simulation system based on multimodal interaction includes an organization simulation module, a multimodal interaction module and an operation trajectory analysis module;

[0005] The tissue simulation module acquires human CT images, calculates the transformation matrix of human tissue surface model points based on the human CT images, and generates a periodic time-resolved dynamic model of human tissue through matrix interpolation.

[0006] The multimodal interaction module uses a method based on positional dynamics and multi-constraint mechanisms to perform biomechanical simulation on a periodic time-resolved dynamic model of human tissues, realizing the real mechanical properties and dynamic interaction of each tissue.

[0007] The operation trajectory analysis module uses a deep learning-based surgical stage recognition method to score the surgical simulation process.

[0008] Furthermore, in the multimodal interaction module, the method based on position dynamics and multiple constraint mechanisms specifically includes the following steps:

[0009] Step A1: Biomechanical modeling of the superficial fascia, using a two-stage solver and triangular mesh to predict the vertex positions of the fascia. The formulas used are as follows: ;

[0010] In the formula, It is the predicted apex location of the fascia. It is the current position of the vertex. It is the velocity of the vertex. It is the time step. It is external acceleration;

[0011] Distance preservation constraints and dihedral angle constraints are used. Distance preservation constraints ensure the tensile strength of the tissue, while dihedral angle constraints control the bending behavior of the tissue. The formulas used are as follows: ; ;

[0012] In the formula, It is the distance preservation constraint formula. It is the dihedral angle constraint formula. It is the length of the particle at rest. , , and These are the four vertices that form two adjacent triangles, among which... For shared edges, and It is the unit normal vector of the adjacent triangle. It is the arctangent function in the fourth quadrant. It is the initial dihedral angle of human tissue;

[0013] Linear damping force and angular damping force are introduced to simulate the viscoelastic properties of fascia. The vertices around the fascia are fixed, and the mass of the vertices of the triangular mesh is evenly distributed to ensure that the deformation behavior conforms to the anatomical law.

[0014] Step A2: IMA vascular biomechanical modeling, using high-stiffness distance preservation constraints and low-stiffness dihedral constraints to balance vascular wall integrity and bending flexibility. High-resolution modeling strategy is used, with low-resolution meshes used for physical calculations and high-resolution meshes updated through linear interpolation to update deformation states, balancing computational efficiency and visualization accuracy.

[0015] A zero-gravity environment was used to recreate the mechanical state of the IMA within the thoracic cavity. The IMA vertex mass was the ratio of the total mass of the blood vessel to the number of fixed points in the low-resolution mesh.

[0016] Step A3: Surgical operation simulation, to achieve accurate simulation of the core surgical actions of cutting, clamping and electrocautery. The topology-preserving cutting algorithm is used to simulate superficial fascia cutting, calculate the signed distance from the fascia vertex to the cutting geometry, filter the vertices of the cutting area, perform local remeshing of the cutting area, generate a new vertex set and edge set, update the triangle network, track the changes of distance preservation constraints and dihedral angle constraints, and synchronously update the current position and reference configuration of the triangle network vertices.

[0017] A connective tissue cutting algorithm was used to simulate adipose tissue cutting. The cutting plane was defined based on the position and vector of the surgical instruments. Connective tissue filaments were discretized into a set of line segments. Cutting criteria were defined; when a line segment met the cutting criteria, it was marked for removal. A visual simulation of the progressive separation of tissue was performed using a particle diameter decay formula. The cutting criteria formula is as follows: ;

[0018] In the formula, It is the first in the line segment set A connective tissue filament, It is the first A point on a connective tissue filament, and These are the orthogonal basis vectors of the orientation of the surgical instruments. These are the current coordinates of the tip of the surgical instrument. It refers to the cutting length of the surgical instrument. It is the normal vector of the cutting plane;

[0019] Step A4: Surgical instrument motion simulation. Model the surgical instrument kinematic chain matrix, calculate the surgical space based on the opening and closing angles of the surgical instruments, and calculate the surgical instrument coordinates by transforming the surgical instrument kinematic chain matrix.

[0020] Step A5: Tactile feedback. The normal resistance simulates the resistance of the surgical instrument penetrating the tissue vertically, and the tangential resistance simulates the resistance of the surgical instrument cutting laterally. The normal resistance and tangential resistance together constitute the tactile feedback force.

[0021] Furthermore, in the operation trajectory analysis module, the surgical stage recognition method based on deep learning specifically includes the following steps:

[0022] Step B1: Data collection, collecting a set of surgical videos, which covers different equipment models, surgical team operating habits and patient baseline characteristics, patient clinical information and surgeon qualification information, and standardizing the surgical videos;

[0023] Step B2: Surgical stage annotation, annotating and dividing the surgical stages frame by frame;

[0024] Step B3: Surgical stage identification. Establish and initialize a convolutional neural network. Input the labeled surgical video set into the convolutional neural network for training. Optimize the hyperparameters of the convolutional neural network through grid search. The goal is to maximize the classification accuracy. Use the weighted average method to calculate the evaluation index to avoid insufficient training of minority classes.

[0025] Step B4: Operation trajectory scoring, including efficiency assessment and quality assessment. The efficiency assessment statistically analyzes the number of frames at each stage output by the convolutional neural network, compares the difference in frame count between the operation trajectory in the surgical video set and the simulated operation trajectory at each stage, screens out key stages sensitive to skill, and verifies the statistical significance of the difference in frame count between the two groups through an independent samples t-test. The assessment criterion is that the fewer the frame count, the better the skill. The quality assessment extracts the feature vectors of the operation trajectory in the surgical video set and the simulated operation trajectory and performs cosine similarity matching to output a similarity score. The significance is verified through a t-test, and the assessment criterion is that the higher the similarity score, the better the quality of the surgical operation trajectory. This achieves a quantitative assessment of the quality of the surgical operation.

[0026] The beneficial effects achieved by the present invention using the above solution are as follows:

[0027] To address the technical problems of low fidelity in replicating the biomechanical properties of human tissues, inaccurate surgical operation simulation, and lack of realistic tactile feedback in existing teaching simulation systems, which result in a lack of clinical realism in surgical operation simulation and an inability to effectively accumulate the muscle memory and judgment experience required for clinical practice, thus greatly affecting the training effect of practical skills, this solution adopts a biomechanical simulation method based on position dynamics and multi-constraint mechanisms. For the superficial fascia, a two-stage solver, distance-preserving constraints, dihedral constraints, and linear and angular damping forces are used for modeling. For the IMA vessels, a combination of high-stiffness distance constraints and low-stiffness dihedral constraints, along with a high- and low-resolution mesh balancing strategy, is employed. For surgical operations, topology-preserving cutting algorithms and connective tissue cutting algorithms are used to simulate core movements. Furthermore, a multi-dimensional tactile feedback mechanism is constructed using normal resistance and tangential resistance. This solution utilizes tactile feedback to accurately replicate the biomechanical properties of different human tissues, providing a realistic tactile interactive experience close to clinical practice. This effectively improves trainees' operational proficiency, standardized movements, and emergency response capabilities. Existing simulation systems suffer from technical problems such as a disconnect between multimodal interaction and operational assessment, insufficient teaching focus, lack of dynamic adaptability, and difficulty in meeting the learning needs of different trainees, resulting in low teaching efficiency and an inability to provide precise empowerment. This solution addresses these issues by employing deep learning algorithms, using grid search to optimize hyperparameters, and calculating assessment indicators using a weighted average method. By comparing the frame rate differences between simulated and standard operations at each stage, and verifying significance using independent samples t-tests, a complete quantitative system for operational trajectory assessment is constructed. This enables precise identification of surgical stages, improving the relevance and efficiency of teaching. Attached Figure Description

[0028] Figure 1 A module connection diagram of an interventional nursing teaching simulation system based on multimodal interaction provided by the present invention;

[0029] Figure 2 The present invention provides a flowchart of a method based on position dynamics and multiple constraint mechanisms.

[0030] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. Detailed Implementation

[0031] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0032] Example 1: See Figure 1This embodiment provides an interventional nursing teaching simulation system based on multimodal interaction. The interventional nursing teaching simulation system based on multimodal interaction includes an organization simulation module, a multimodal interaction module, and an operation trajectory analysis module.

[0033] The tissue simulation module acquires human CT images, calculates the transformation matrix of human tissue surface model points based on the human CT images, and generates a periodic time-resolved dynamic model of human tissue through matrix interpolation.

[0034] The multimodal interaction module uses a method based on positional dynamics and multi-constraint mechanisms to perform biomechanical simulation on a periodic time-resolved dynamic model of human tissues, realizing the real mechanical properties and dynamic interaction of each tissue.

[0035] The operation trajectory analysis module uses a deep learning-based surgical stage recognition method to score the surgical simulation process.

[0036] Example 2: See Figure 1 and Figure 2 This embodiment, based on the above embodiment, specifically includes the following steps in the multimodal interaction module: A method based on position dynamics and multiple constraint mechanisms:

[0037] Step A1: Biomechanical modeling of the superficial fascia, using a two-stage solver and triangular mesh to predict the vertex positions of the fascia. The formulas used are as follows: ;

[0038] In the formula, It is the predicted apex location of the fascia. It is the current position of the vertex. It is the velocity of the vertex. It is the time step. It is external acceleration;

[0039] Distance preservation constraints and dihedral angle constraints are used. Distance preservation constraints ensure the tensile strength of the tissue, while dihedral angle constraints control the bending behavior of the tissue. The formulas used are as follows: ; ;

[0040] In the formula, It is the distance preservation constraint formula. It is the dihedral angle constraint formula. It is the length of the particle at rest. , , and These are the four vertices that form two adjacent triangles, among which... For shared edges, and It is the unit normal vector of the adjacent triangle. It is the arctangent function in the fourth quadrant. It is the initial dihedral angle of human tissue;

[0041] Linear damping force and angular damping force are introduced to simulate the viscoelastic properties of fascia. The vertices around the fascia are fixed, and the mass of the vertices of the triangular mesh is evenly distributed to ensure that the deformation behavior conforms to the anatomical law.

[0042] Step A2: IMA vascular biomechanical modeling, using high-stiffness distance preservation constraints (stiffness coefficient is 2 times that of fascia) and low-stiffness dihedral constraints (stiffness coefficient is 0.5 times that of fascia), taking into account both the integrity of the vessel wall and bending flexibility. High-resolution modeling strategy is used, with low-resolution mesh used for physical calculations and high-resolution mesh updated through linear interpolation to balance computational efficiency and visualization accuracy.

[0043] A zero-gravity environment was used to recreate the mechanical state of the IMA within the thoracic cavity. The IMA vertex mass was the ratio of the total mass of the blood vessel to the number of fixed points in the low-resolution mesh.

[0044] Step A3: Surgical operation simulation, to achieve accurate simulation of the core surgical actions of cutting, clamping and electrocautery. The topology-preserving cutting algorithm is used to simulate superficial fascia cutting, calculate the signed distance from the fascia vertex to the cutting geometry, filter the vertices of the cutting area, perform local remeshing of the cutting area, generate a new vertex set and edge set, update the triangle network, track the changes of distance preservation constraints and dihedral angle constraints, and synchronously update the current position and reference configuration of the triangle network vertices.

[0045] A connective tissue cutting algorithm was used to simulate adipose tissue cutting. The cutting plane was defined based on the position and vector of the surgical instruments. Connective tissue filaments were discretized into a set of line segments. Cutting criteria were defined; when a line segment met the cutting criteria, it was marked for removal. A visual simulation of the progressive separation of tissue was performed using a particle diameter decay formula. The cutting criteria formula is as follows: ;

[0046] In the formula, It is the first in the line segment set A connective tissue filament, It is the first A point on a connective tissue filament, and These are the orthogonal basis vectors of the orientation of the surgical instruments. These are the current coordinates of the tip of the surgical instrument. It refers to the cutting length of the surgical instrument. It is the normal vector of the cutting plane;

[0047] Step A4: Surgical instrument motion simulation. Model the surgical instrument kinematic chain matrix, calculate the surgical space based on the opening and closing angles of the surgical instruments, and calculate the surgical instrument coordinates by transforming the surgical instrument kinematic chain matrix.

[0048] Step A5: Tactile feedback. The normal resistance simulates the resistance of the surgical instrument penetrating the tissue vertically, and the tangential resistance simulates the resistance of the surgical instrument cutting laterally. The normal resistance and tangential resistance together constitute the tactile feedback force.

[0049] Example 3: See Figure 1 This embodiment is based on the above embodiment. In the operation trajectory analysis module, the surgical stage recognition method based on deep learning specifically includes the following steps:

[0050] Step B1: Data collection, collecting a set of surgical videos, which covers different equipment models, surgical team operating habits and patient baseline characteristics, patient clinical information and surgeon qualification information, and standardizing the surgical videos;

[0051] Step B2: Surgical stage annotation, annotating and dividing the surgical stages frame by frame;

[0052] Step B3: Surgical stage identification. Establish and initialize a convolutional neural network. Input the labeled surgical video set into the convolutional neural network for training. Optimize the hyperparameters of the convolutional neural network through grid search. The goal is to maximize the classification accuracy. Use the weighted average method to calculate the evaluation index to avoid insufficient training of minority classes.

[0053] Step B4: Operation trajectory scoring, including efficiency assessment and quality assessment. The efficiency assessment statistically analyzes the number of frames at each stage output by the convolutional neural network, compares the difference in frame count between the operation trajectory in the surgical video set and the simulated operation trajectory at each stage, screens out key stages sensitive to skill, and verifies the statistical significance of the difference in frame count between the two groups through an independent samples t-test. The assessment criterion is that the fewer the frame count, the better the skill. The quality assessment extracts the feature vectors of the operation trajectory in the surgical video set and the simulated operation trajectory and performs cosine similarity matching to output a similarity score. The significance is verified through a t-test, and the assessment criterion is that the higher the similarity score, the better the quality of the surgical operation trajectory. This achieves a quantitative assessment of the quality of the surgical operation.

[0054] Example 4: This example is based on the above examples. In Example 2, IMA vascular biomechanical modeling uses high-stiffness distance-preserving constraints and low-stiffness dihedral constraints. The stiffness coefficient of the high-stiffness distance-preserving constraints is twice that of the fascia, and the stiffness coefficient of the low-stiffness dihedral constraints is 0.5 times that of the fascia.

[0055] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0056] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

[0057] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. A multimodal interactive interventional nursing teaching simulation system, characterized in that, It includes an organization simulation module, a multimodal interaction module, and an operation trajectory analysis module; The tissue simulation module acquires human CT images, calculates the transformation matrix of human tissue surface model points based on the human CT images, and generates a periodic time-resolved dynamic model of human tissue through matrix interpolation. The multimodal interaction module uses a method based on position dynamics and multi-constraint mechanisms to perform biomechanical simulation on a periodic time-resolved dynamic model of human tissue. The operation trajectory analysis module uses a deep learning-based surgical stage recognition method to score the surgical simulation process.

2. The interventional nursing teaching simulation system based on multimodal interaction according to claim 1, characterized in that, In the multimodal interaction module, the method based on position dynamics and multiple constraint mechanisms specifically includes the following steps: Step A1: Biomechanical modeling of the superficial fascia, using a two-stage solver and triangular mesh to predict the vertex positions of the fascia; Distance-maintaining constraints and dihedral angle constraints are used to ensure the tensile strength of the tissue and control its bending behavior; Linear damping force and angular damping force are introduced to fix the vertices around the fascia. The mass of the vertices of the triangular mesh is evenly distributed to ensure that the deformation behavior conforms to the anatomical law. Step A2: IMA vascular biomechanical modeling, using high-stiffness distance preservation constraints and low-stiffness dihedral constraints, and utilizing a high-resolution modeling strategy to balance computational efficiency and visualization accuracy; A zero-gravity environment was used to recreate the mechanical state of the IMA within the thoracic cavity. The IMA vertex mass was the ratio of the total mass of the blood vessel to the number of fixed points in the low-resolution mesh. Step A3: Surgical operation simulation. The topology-preserving cutting algorithm is used to simulate superficial fascia cutting. The signed distance from the fascia vertex to the cutting geometry is calculated, the vertices of the cutting region are filtered, the cutting region is locally remeshed, a new vertex set and edge set are generated, the triangle network is updated, the changes of distance preservation constraints and dihedral angle constraints are tracked, and the current position and reference configuration of the triangle network vertices are updated synchronously. A connective tissue cutting algorithm is used to simulate adipose tissue cutting. The cutting plane is defined based on the position and vector of the surgical instrument. The connective tissue filaments are discretized into a set of line segments. Cutting judgment conditions are defined. When a line segment meets the cutting judgment conditions, it is marked as removed. The progressive separation of tissue is simulated by the particle diameter decay formula for visual simulation. Step A4: Surgical instrument motion simulation. Model the surgical instrument kinematic chain matrix, calculate the surgical space based on the opening and closing angles of the surgical instruments, and calculate the surgical instrument coordinates by transforming the surgical instrument kinematic chain matrix. Step A5: Tactile feedback. The normal resistance simulates the resistance of the surgical instrument penetrating the tissue vertically, and the tangential resistance simulates the resistance of the surgical instrument cutting laterally. The normal resistance and tangential resistance together constitute the tactile feedback force.

3. The interventional nursing teaching simulation system based on multimodal interaction according to claim 2, characterized in that, In the operation trajectory analysis module, the surgical stage recognition method based on deep learning specifically includes the following steps: Step B1: Data collection, acquiring a set of surgical videos, and standardizing the surgical videos; Step B2: Surgical stage annotation, annotating and dividing the surgical stages frame by frame; Step B3: Surgical stage identification. Establish and initialize a convolutional neural network. Input the labeled surgical video set into the convolutional neural network for training. Optimize the hyperparameters of the convolutional neural network through grid search. The goal is to maximize the classification accuracy. Use the weighted average method to calculate the evaluation index to avoid insufficient training of minority classes. Step B4: Operation trajectory scoring, including efficiency assessment and quality assessment. The efficiency assessment statistically analyzes the number of frames at each stage output by the convolutional neural network, compares the difference in frame count between the operation trajectory in the surgical video set and the simulated operation trajectory at each stage, screens out key stages sensitive to skill, and verifies the statistical significance of the difference in frame count between the two groups through an independent samples t-test. The assessment criterion is that the fewer the frame count, the better the skill. The quality assessment extracts the feature vectors of the operation trajectory in the surgical video set and the simulated operation trajectory and performs cosine similarity matching to output a similarity score. The significance is verified through a t-test, and the assessment criterion is that the higher the similarity score, the better the quality of the surgical operation trajectory. This achieves a quantitative assessment of the quality of the surgical operation.

Images