Craniocerebral operation training method and system based on digital twinning

Abstract

The invention discloses a craniocerebral operation training method and system based on digital twinning, and relates to the technical field of medical training, and the method comprises the steps: building a digital twinning model, and constructing a virtual interaction scene; carrying out key frame extraction on the interaction image of the virtual interaction scene, carrying out user identification operation on the key frame image, obtaining operation information of a user in a training process, evaluating a training result of the user, and formulating a training scheme. Through preoperative simulation, a nurse can identify and avoid risks possibly occurring in an operation in advance, so that the operation safety and the training efficiency of the operating room instrument nurse are improved; a highly real and interactive operation simulation environment is provided for nurses, personalized training suggestions can be provided according to actual operation performance of the nurses, and the operation cooperation effect of the nurses is improved. Modules for training stages or training different stages can be arranged, and the instrument nurses can learn the content of a certain stage in a reinforced mode according to the learning progress during training, so that the learning effect is consolidated.

Description

Technical Field

[0001] The present invention relates to the field of medical training technology, and more particularly to a cranial surgery training method and system based on digital twins. Background Art

[0002] Cranial surgery is a high-risk, high-tech medical operation that places extremely high demands on the professional skills and surgical experience of medical staff. Traditional training methods rely on surgical observation and one-on-one teaching. One-on-one teaching means that experienced nurses teach new nurses. However, for new nurses, due to lack of mastery of basic surgical steps, lack of proficiency in the operation of surgical instruments and equipment, incomplete awareness and operation techniques of aseptic operation, there are risks of aseptic operation, which affects the use of instruments and equipment, affects the progress of surgery, and slows down the teaching process. At the same time, the learning pressure on novice nurses is huge. In addition, due to the different surgical operation habits of different doctor groups, it will make teaching more difficult and lack personalized and interactive training methods.

[0003] With the development of digital twin and virtual reality technology, preoperative simulation and surgical planning can be achieved by creating an accurate digital copy of the patient's anatomy, thereby improving surgical safety and the training efficiency of operating room instrument nurses.

[0004] Therefore, how to propose a cranial surgery training method and system based on digital twins to achieve preoperative simulation and surgical planning, thereby improving surgical safety and the training efficiency of operating room instrument nurses is an urgent problem that technical personnel in this field need to solve. Summary of the invention

[0005] In view of this, the present invention provides a cranial surgery training method and system based on digital twins to realize preoperative simulation and surgical planning, thereby improving surgical safety and the training efficiency of operating room instrument nurses. In order to achieve the above purpose, the present invention adopts the following technical solutions:

[0006] A cranial surgery training method based on digital twins, comprising:

[0007] Collecting entity data of the training surgical instrument, generating point cloud data of the training surgical instrument according to the entity data, and performing three-dimensional reconstruction on the point cloud data to generate a three-dimensional reconstruction model of the training surgical instrument;

[0008] A digital twin model is established based on the three-dimensional reconstruction model of the training surgical instrument, and a virtual interactive scene is constructed based on the digital twin model;

[0009] Extract key frames from interactive images of virtual interactive scenes, identify user operations on key frame images, obtain user operation information during training, and evaluate user training results based on the operation information;

[0010] Develop training plans based on users' training results.

[0011] Optionally, the three-dimensional reconstruction of the point cloud data to generate a three-dimensional reconstruction model of the training surgical instrument includes:

[0012] Acquire multi-angle image information of the training surgical instrument, acquire point cloud data based on the multi-angle image information, pre-process the point cloud data, and acquire the size and assembly information of the training surgical instrument;

[0013] Perform 3D reconstruction through point cloud data, align and connect the point cloud data according to size and assembly information, and generate a 3D reconstruction model of the training surgical instrument;

[0014] Assign the material properties, electromechanical object properties and motion constraints of the training surgical instrument to the 3D reconstruction model to build a digital twin model of the training surgical instrument;

[0015] Motion-match the digital twin model of the training surgical instrument to the training surgical instrument.

[0016] Optionally, obtaining operation information of the user during the training process includes:

[0017] Establish the connection and mapping relationship between the digital twin model and the training surgical instrument, and perform data transmission of the digital twin model;

[0018] A virtual interactive scene is constructed through the digital twin model of the training surgical instrument, and relevant training teaching videos are preset in the virtual interactive scene to guide the user's operation;

[0019] Key frame extraction is performed on the interactive image of the virtual interactive scene obtained by data transmission, and the user's operation is identified through the key frame image.

[0020] Optionally, the operation of identifying the user includes:

[0021] The user's operation is divided into the user's hand movements and the training surgical instrument selected by the user;

[0022] Matching the image features of the key frame image with the image features of the pre-stored hand movements, and taking the type of the hand movement corresponding to the highest similarity as the type of the user's hand movement;

[0023] Matching the image features of the training surgical instrument in the key frame image with the image features of the pre-stored training surgical instrument, and taking the type of the training surgical instrument corresponding to the highest similarity as the type of the training surgical instrument selected by the user;

[0024] The key frame image features are matched with the pre-stored operation image features, and the type of operation corresponding to the highest similarity is identified as the type of user operation.

[0025] Optionally, it also includes: when there is a deviation between the user's operation and the operation in the teaching video, the operation is marked; at the same time, corresponding motion variables are generated according to the operation of the digital twin model, and motion extreme values ​​are generated by training the motion connection and motion constraints of each component in the surgical instrument. When the motion variable is greater than the motion extreme value, it is marked.

[0026] Optionally, evaluating the user's training results according to the operation information includes:

[0027] Obtain the user's training completion time, preset the standard duration of each operation according to the complexity of the operation, characterize the user's proficiency through the actual duration of each operation completed by the user, calculate the user's average proficiency during training based on the training completion time, and use the average proficiency as the proficiency benchmark to obtain the operation steps in which the user's proficiency during training is lower than the proficiency benchmark;

[0028] The user training results are evaluated through the time information of the user's completion, the number of times marked, the number of operation steps with proficiency lower than the proficiency benchmark, and the proportion of the operation steps to the time information of the user's completion.

[0029] Optionally, formulating a training plan according to the training results of the user includes:

[0030] Obtain the operation of training surgical instruments and user training results, and vectorize the operations;

[0031] Obtain information on the number of times each operation is marked and the number of times the proficiency is determined to be lower than the proficiency benchmark from the user training results, and generate an attention score for each operation based on the number information;

[0032] Generate an attention-based vectorized representation of each operation in the user training result through the attention score and the vectorized representation of the operation;

[0033] Train and build a training plan recommendation model, input the attention-based vectorized representation of each operation in the user training results into the training plan recommendation model, and generate a recommended training plan.

[0034] Optionally, the training and building of a training program recommendation model includes:

[0035] Obtaining the training results of the operation of the surgical instrument and the user, as well as the vectorized representation of each operation in the user training results based on the attention mechanism, performing data preprocessing, and obtaining a training set;

[0036] A training plan recommendation model is constructed based on a neural network with an attention mechanism, and the training is initialized using a training set. A classification model is constructed using a multilayer perceptron with two fully connected layers to recommend training plans. The relu function is selected as the activation function, and the output value is converted into a probability through the sigmoid function. The training effect is updated by coupling the knowledge state to obtain the final recommended training plan.

[0037] Optionally, a cranial surgery training system based on digital twins, comprising:

[0038] Collection module: used to collect physical data of training surgical instruments;

[0039] 3D reconstruction module: used to generate point cloud data of training surgical instruments based on entity data, and to perform 3D reconstruction on the point cloud data to generate a 3D reconstruction model of the training surgical instruments;

[0040] Interaction module: used to establish a digital twin model based on the 3D reconstruction model of the training surgical instrument, and to build a virtual interaction scene based on the digital twin model;

[0041] Recognition module: used to extract key frames from interactive images of virtual interactive scenes, identify user operations on key frame images, and obtain user operation information during the training process;

[0042] Evaluation module: used to evaluate the user's training results based on the operation information;

[0043] Development module: used to develop training plans based on the user's training results.

[0044] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a cranial surgery training method and system based on digital twins, which has the following beneficial effects:

[0045] The present invention proposes a cranial surgery training method based on digital twins, including: collecting entity data of training surgical instruments, generating point cloud data of training surgical instruments according to the entity data, performing three-dimensional reconstruction on the point cloud data to generate a three-dimensional reconstruction model of the training surgical instruments; establishing a digital twin model according to the three-dimensional reconstruction model of the training surgical instruments, and constructing a virtual interactive scene according to the digital twin model; extracting key frames from the interactive images of the virtual interactive scene, identifying the user's operation on the key frame images, obtaining the user's operation information during the training process, and evaluating the user's training results according to the operation information; and formulating a training plan according to the user's training results. The present invention realizes preoperative simulation and surgical planning. Through preoperative simulation, nurses can pre-identify and avoid possible risks during surgery, thereby improving surgical safety and the training efficiency of operating room instrument nurses; providing nurses with a highly realistic and interactive surgical simulation environment, and providing personalized training suggestions based on the actual operation performance of the nurses, thereby improving the surgical cooperation effect of the nurses. Training stages or modules for different training stages can also be set. During the training of instrument nurses, the content of a certain stage can be strengthened according to the learning progress to consolidate the learning effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0047] Figure 1 A schematic flow chart of a cranial surgery training method based on digital twins provided by the present invention.

[0048] Figure 2 A structural framework diagram of a cranial surgery training system based on digital twins provided by the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] The embodiment of the present invention discloses a cranial surgery training method based on digital twins, such as Figure 1 As shown, including:

[0051] Collecting entity data of the training surgical instrument, generating point cloud data of the training surgical instrument according to the entity data, and performing three-dimensional reconstruction on the point cloud data to generate a three-dimensional reconstruction model of the training surgical instrument;

[0052] A digital twin model is established based on the three-dimensional reconstruction model of the training surgical instrument, and a virtual interactive scene is constructed based on the digital twin model;

[0053] Extract key frames from interactive images of virtual interactive scenes, identify user operations on key frame images, obtain user operation information during training, and evaluate user training results based on the operation information;

[0054] Develop training plans based on users' training results.

[0055] Furthermore, the training surgical instrument comprises:

[0056] Simple instruments, that is, instruments that do not require assembly or adjustment, include:

[0057] Wire scissors, meningeal scissors, microsurgery scissors, bipolar and monopolar electrosurgery, suction device, scalp clamp, bone peeler, nerve stripper, bone pryer, bone rongeur, meningeal hook needle, brain pressure plate, vascular clamp, tumor removal forceps, needle holder;

[0058] For the above instruments, nurses need to pay attention to whether the instrument transfer technique is standardized, whether the transfer is stable, accurate, light and fast, whether it can be transferred and used in a timely and accurate manner according to the surgical steps and needs, and pay attention to checking the cleanliness and integrity of the instruments in a timely manner, and recycle the instruments in time when not in use and place them properly.

[0059] Devices that require assembly inspection include:

[0060] Dermatomes, sharp knives, imported head collars, craniotomy drills (electric or pneumatic), ultrasonic scalpels, etc. require attention to the correct assembly and intact functions of the instruments and equipment, and the inspection, use and disassembly should be in accordance with the operating procedures.

[0061] Furthermore, the three-dimensional reconstruction of the point cloud data to generate a three-dimensional reconstruction model of the training surgical instrument includes:

[0062] Acquire multi-angle image information of the training surgical instrument, acquire point cloud data based on the multi-angle image information, pre-process the point cloud data, and acquire the size and assembly information of the training surgical instrument;

[0063] Perform 3D reconstruction through point cloud data, align and connect the point cloud data according to size and assembly information, and generate a 3D reconstruction model of the training surgical instrument;

[0064] Assign the material properties, electromechanical object properties and motion constraints of the training surgical instrument to the 3D reconstruction model to build a digital twin model of the training surgical instrument;

[0065] Motion-match the digital twin model of the training surgical instrument to the training surgical instrument.

[0066] Specifically, the surgical instrument is built with a GPS module and a pressure sensor module, and the information obtained through the GPS module and the pressure sensor module is added to the three-dimensional reconstruction model.

[0067] Furthermore, the obtaining of the user's operation information during the training process includes:

[0068] Establish the connection and mapping relationship between the digital twin model and the training surgical instrument, and perform data transmission of the digital twin model;

[0069] A virtual interactive scene is constructed through the digital twin model of the training surgical instrument, and relevant training teaching videos are preset in the virtual interactive scene to guide the user's operation;

[0070] Key frame extraction is performed on the interactive image of the virtual interactive scene obtained by data transmission, and the user's operation is identified through the key frame image.

[0071] Furthermore, medical imaging data of patients, such as MRI and CT scans, are collected, and a three-dimensional digital model of the patient's brain is constructed using the collected data to form a digital twin. The digital twin is mapped to the digital twin model, and relevant training teaching videos are preset in the virtual interactive scene to guide users' operations on the digital twin.

[0072] Furthermore, the operation of identifying the user includes:

[0073] The user's operation is divided into the user's hand movements and the training surgical instrument selected by the user;

[0074] Matching the image features of the key frame image with the image features of the pre-stored hand movements, and taking the type of the hand movement corresponding to the highest similarity as the type of the user's hand movement;

[0075] Matching the image features of the training surgical instrument in the key frame image with the image features of the pre-stored training surgical instrument, and taking the type of the training surgical instrument corresponding to the highest similarity as the type of the training surgical instrument selected by the user;

[0076] The key frame image features are matched with the pre-stored operation image features, and the type of operation corresponding to the highest similarity is identified as the type of user operation.

[0077] Specifically, the types of hand movements of the user include: standard movements of instrument transfer methods, involving hand movements and transfer techniques, and the image features of the pre-stored hand movements are standard movement image features of instrument transfer methods. The purpose is to enable users to accurately and quickly transfer according to surgical steps through training on standard movements of instrument transfer methods, pay attention to emergency handling, achieve tacit cooperation with the surgical doctor team, and achieve professional focus.

[0078] In a specific embodiment, the standard action standard of the instrument delivery method is:

[0079] 1) Fast speed, accurate method, and correct instrument. The operator can use it without adjusting the direction after taking it;

[0080] 2) The force should be appropriate to remind the operator's attention;

[0081] 3) Adjust surgical instruments in time according to the surgical site;

[0082] 4) Retract the instruments around the incision in time to avoid accumulation and prevent them from falling to the ground;

[0083] 5) When holding the instrument, the curved side should be facing upwards, the handle should be facing the surgeon, single-sided instruments should be handed vertically, and sharp instruments should be handed horizontally with the blade facing downwards;

[0084] 6) The integrity of the instrument should be checked before and after delivery to prevent missing parts from being left in the surgical site;

[0085] 7) Soak the retractor in saline before transferring it;

[0086] 8) When installing or removing the blade, be careful to avoid people and keep the tip downward, facing the sterile instrument table;

[0087] 9) When delivering sharp instruments, it is recommended to use non-contact delivery to prevent occupational exposure;

[0088] 10) Pass the instrument to the opposite side by leaping over it; it is prohibited to pass it over the doctor’s shoulder or behind his back.

[0089] In a specific embodiment, the standard actions of the device delivery method specifically include:

[0090] 1. Surgical knife delivery method: Be careful not to hurt yourself and the surgeon. There are two methods of delivering the knife: the same-side delivery method and the opposite-side delivery method. When delivering the scalpel, hold the back of the knife with the blade facing down and the tip facing backward and deliver it horizontally; 2) Installing the blade: Hold the back side of the front end of the blade with a needle holder, and gently align the blade with the handle slot; 3) Removing the blade: Hold the back side of the rear end of the blade with a needle holder, lift it upwards slightly, and push it out of the handle slot; 4) Delivering the scalpel: Use a curved disc for contactless delivery, and deliver it horizontally to the surgeon to prevent occupational exposure.

[0091] 2. Tweezers delivery method

[0092] 1) Hold the tip of the forceps, close the opening, and pass it upright;

[0093] 2) In case of emergency during surgery, the thumb, index finger and middle finger can be used to hold the tail of the forceps, and the open end of the forceps can be closed with the combined force of three fingers, allowing the surgeon to hold the middle part of the forceps.

[0094] (1) Contralateral transfer method: The thumb of the right hand holds the upper third of the convex side, and the four fingers hold the middle of the concave side. Through appropriate wrist movements, the handle ring of the instrument is tapped on the palm of the operator.

[0095] (2) Same-side transmission method: The thumb and ring finger of the right hand hold the concave side, and the index finger and middle finger hold the upper third of the convex side, and transmit through the wrist. The left hand does the opposite.

[0096] (3) Cross-transfer method: When passing two instruments at the same time, the hand passing the opposite instrument is on top and the hand passing the same instrument is on the bottom. The instrument cannot be passed from the operator's shoulder or back.

[0097] 3. Scissors passing method

[0098] The right thumb holds the upper third of the convex side, and the four fingers hold the middle of the concave side, with the arc of the scissors facing the nurse's palm. Through appropriate wrist movements, the handle ring of the instrument is slapped on the operator's palm.

[0099] 4. How to add needle to the needle holder

[0100] The front 1 / 3 of the needle holder opening clamps the back 1 / 3 of the needle, and the suture is inserted into the front 1 / 3 of the needle holder.

[0101] 5. Needle holder delivery method

[0102] Holding the needle: hold with the right hand; clamping the needle: the front 1 / 3 of the needle holder opening clamps the back 1 / 3 of the suture; passing: the nurse holds the middle of the needle holder with the tip of the needle facing the palm, the needle arc facing the back of the hand, the suture is placed on the back of the hand or held in the palm, and the wrist is used to slap the handle ring on the operator's palm with appropriate force.

[0103] 6. Hook transfer method

[0104] Before passing the retractor, soak it in saline. Hold the front end of the retractor and pass the handle end parallel to the retractor.

[0105] 7. Rongeur delivery method

[0106] Spear-shaped rongeur is passed by holding the shaft and the hand connects to the handle. Double-link rongeur is passed by holding the head end and the hand connects to the handle.

[0107] 8. Principles of Dressing Delivery

[0108] Fast speed, accurate method, correct items, no debris or debris; timely change of incision dressings to avoid accumulation; gauze dressings should be opened, soaked, and passed at an angle. One end of the fixing belt or gauze should be left at the incision. Do not insert it completely into the body cavity to avoid being left in the tissue. 1) Gauze and gauze pad delivery method: open the gauze and gauze pad, and the washed nurse holds the two ends of the gauze and gauze pad with both hands and passes them at an angle; 2) Drainage tube delivery: often used for tissue protective traction, the curved vascular clamp clamps the head end and hands it to the operator. After folding the drainage tube, fix it with a small straight clamp; 3) Rubber band delivery: fingers stretch the rubber ring and put it on the operator's right hand, which is used for cluster fixation or tissue traction of multiple vascular clamps; 4) Brain cotton pad delivery: mostly used in craniotomy, the cotton pad is placed on the tissue surface for protective suction. One end of the brain cotton pad is required to have a developing line to avoid being left behind. Pull slightly to check the quality of the brain cotton piece. After soaking it, use the index finger to support it and the operator to hold one end of the cotton piece.

[0109] In a specific implementation, matching the key frame image features with the pre-stored operation image features and identifying the type of operation corresponding to the highest similarity as the type of user operation also includes:

[0110] The first key frame image and the second key frame image are identified to obtain the type of the key frame operated by the user. The specific steps are:

[0111] Perform image segmentation processing on the first key frame image and the second key frame image respectively, and obtain a first set of areas to be filtered in the first key frame image and a set of areas to be filtered in the second key frame image; obtain a third set of heights of the areas to be filtered in the first set; obtain a fourth set of heights of the areas to be filtered in the second set; obtain the intersection of the third set and the fourth set; obtain a fifth set of areas to be filtered in the first set whose heights belong to the intersection; perform feature extraction on the areas to be filtered in the fifth set, and obtain image features of each area to be filtered in the fifth set; match the image features of each area to be filtered in the fifth set with the image features of pre-stored key frames of various types of operations, and use the type of key frame of the operation corresponding to the highest similarity as the type of key frame of the user's operation.

[0112] Furthermore, it also includes: when there is a deviation between the user's operation and the operation in the teaching video, the operation is marked; at the same time, the corresponding motion variables are generated according to the operation of the digital twin model, and the motion extreme values ​​are generated by training the motion connection and motion constraints of each component in the surgical instrument. When the motion variable is greater than the motion extreme value, it is marked.

[0113] In a specific implementation, it also includes: performing state rollback when the user makes an erroneous operation through a mapping relationship between the user's operation and the movement of the digital twin model:

[0114] An operation list of training surgical instruments is obtained according to a preset teaching video, and the motion information of the corresponding training surgical instruments is matched according to the operation list, and a motion feature table is generated from the motion information; a mapping relationship is generated between each operation in the motion feature table and the position information of each component of the corresponding training surgical instrument, and the mapping relationship is represented by a hash table, and a digital twin model of the training surgical instrument is imported; when a warning message appears and the user needs to roll back the operation, a search is performed in the hash table header according to the current operation, and after finding the hash table header corresponding to the operation, a hash table node connected to the hash table header is obtained; the user is provided with a choice of rollback based on the position information of each component in the hash table node; at the same time, during the user training process, the user searches and intercepts in the hash table according to the unskilled operation, and the intercepted hash table fragment is spliced ​​with the head node and the tail node to realize repeated training of the operation scenario.

[0115] Furthermore, the evaluating the user's training results according to the operation information includes:

[0116] Obtain the user's training completion time, preset the standard duration of each operation according to the complexity of the operation, characterize the user's proficiency through the actual duration of each operation completed by the user, calculate the user's average proficiency during training based on the training completion time, and use the average proficiency as the proficiency benchmark to obtain the operation steps in which the user's proficiency during training is lower than the proficiency benchmark;

[0117] The user training results are evaluated through the time information of the user's completion, the number of times marked, the number of operation steps with proficiency lower than the proficiency benchmark, and the proportion of the operation steps to the time information of the user's completion.

[0118] Furthermore, the evaluation of user training results includes: presetting a user training evaluation system, obtaining the user's training result score, and generating the user's usual score based on the user's training result score; at the same time, according to the average proficiency of the user's current training operation and the operation steps with proficiency lower than the proficiency benchmark, the extracted features are learned through machine learning algorithms such as multi-layer perceptron (MLP), support vector machine (SVM), random forest, deep learning, etc., to establish a prediction model, predict its performance in actual surgery, and predict the user's surgical cooperation degree. Provide real-time feedback during the simulated surgery, including the interaction effect between the surgical tool and the digital twin and possible complications.

[0119] In a specific implementation, the user training evaluation system includes:

[0120] 1. Skin incision: (1) Preparation of items: square plate (round knife), 2 toothed forceps, 2 gauze, scalp clamp, scalp hemostatic clip; (2) Operation evaluation: ① Failure to pay attention to the progress of the operation in a timely manner and effectively transfer and recover bipolar forceps, toothed forceps, scalp clamps and other instruments; ② Failure to put knives and other sharp objects on the tray for transfer, and direct transfer of sharp objects between doctors and nurses;

[0121] 2. Free flap and periosteum stripping: (1) Preparation of items: monopolar electrosurgical unit, bipolar electrocoagulation unit, osteophyte, gauze, imported head ring; Operation evaluation: ① Incomplete or untimely preparation of items; ② Failure to pay attention to the progress of the operation in a timely manner and effectively transfer and recover bipolar, monopolar, toothed forceps, osteophyte, imported head ring, gauze and other instruments and items; ③ Failure to remind the doctor to remove the monopolar smoke in a timely manner;

[0122] 3. Bone flap formation: (1) Preparation of items; craniotomy air drill or electric drill, nerve, dissector, bone wax, bone prying device, irrigator, aspirator, bone rongeur, gauze; Operation evaluation: ① Failure to pay attention to the progress of the operation in a timely manner and effectively transfer and retrieve bipolar, craniotomy equipment, bone prying device, nerve dissector, bone wax, bone prying device, irrigator, aspirator, bone rongeur, gauze and other instruments and items; ② Failure to timely and accurately transfer bone wax, sponge, shorthand gauze, gauze and other hemostatic materials; ③ Failure to timely transfer appropriate nerve dissector according to the surgeon's surgical habits; ④ Failure to adjust the bipolar power in a timely manner; ⑤ Failure to timely transfer bone rongeur and gauze according to surgical needs;

[0123] 4. Epidural hemostasis and preparation of microscope: (1) Preparation of items: grinding drill bit, concave-convex brain pressure plate, medium suction head, irrigator, 4-0 absorbable suture, meningeal forceps, meningeal scissors, wide cotton strips, thread cotton strips, head pad, towel clamp, sterile microscope protective cover, sterile rubber band, tissue scissors; Operation evaluation: ① Failure to pay attention to the progress of the operation in time and effectively transfer and recover instruments and items such as grinding drill bit, concave-convex brain pressure plate, medium suction head, etc.; ② Failure to replace the medium suction head in time; ③ Failure to prepare small hole equipment in time and accurately according to different skull drilling equipment; ④ Failure to use concave-convex brain pressure plate to protect the dura mater according to the doctor's operating habits; ⑤ Failure to correctly prepare 4-0 absorbable suture in advance; ⑥ Failure to correctly prepare the dura mater suspension instrument in advance; ⑦ Failure to prepare the microscope equipment in time according to the microscope cover installation operating procedures; ⑧ Failure to prepare wide cotton strips, wet gauze, head pad, towel clamp and other items according to the surgeon's habits;

[0124] 5. Cutting and pulling the dura mater: (1) Preparation of items: bipolar electrocoagulation, microsuction tips, meningeal hooks, sharp knives, meningeal scissors, and meningeal forceps; Operation evaluation: ① Failure to pay attention to the progress of the operation in a timely manner and effectively transfer and recover bipolar, microsuction tips, meningeal hooks, sharp knives, meningeal scissors, meningeal forceps and other instruments; ② Failure to use the indirect transfer method to transfer and recover meningeal hooks, sharp knives and sharp objects; ③ Failure to prepare needle threading materials in a timely and accurate manner; ④ Failure to prepare suspension sutures and patterned vascular clamps in a timely and correct manner according to the surgeon's habits;

[0125] 6. Exposure and removal of tumors: (1) Preparation of items: bipolar electrocoagulation, microsuction tips, microscissors, serpentine fixation buttons, serpentine retractors, brain compression plates, tumor removal forceps, medicine cups, threads, and cotton strips; Operation evaluation: ① Failure to pay attention to the progress of the operation in a timely manner and effectively transfer and recover bipolar, microscissors, serpentine retractors, brain compression plates, thread cotton strips and other instruments and items; ② Failure to select a suitable brain compression plate according to surgical needs and dip it in water for use; ③ Failure to use the correct method to retain and preserve frozen specimens; ④ Failure to transfer microscissors, bipolar and other instruments and items in a timely manner according to the progress of the operation; ⑤ Failure to accurately cut the appropriate size of cotton strips and transfer them correctly; ⑥ Failure to pay attention to the number of cotton strips used and the integrity of the cotton strips; ⑦ Failure to collect tumor specimens in a timely manner and failure to use standardized isolation techniques to isolate tumors and lesions;

[0126] 7. Tumor cavity hemostasis: (1) Preparation of items: bipolar electrocoagulation, micro suction tip, tumor removal forceps (small), and short-term gauze; Operation evaluation: ① Failure to pay attention to the progress of the operation in a timely manner and effectively transfer and recover bipolar electrocoagulation, tumor removal forceps, short-term gauze and other instruments and items; ② Failure to correctly cut and transfer short-term gauze and other hemostatic products according to the needs of the surgeon and the needs of the surgical area;

[0127] 8. Closing the dura mater: (1) Preparation of items: artificial dura mater, meningeal scissors, meningeal forceps, absorbable suture 4-0, bipolar electrocoagulation, medium-sized suction head, hemostatic materials (fluid gelatin / collagen hemostatic agent); Operation evaluation: ① Failure to pay attention to the progress of the operation in a timely manner and effectively deliver bipolar, suction, hemostatic materials, artificial dura mater and other instruments and items; ② Failure to prepare hemostatic products such as fluid gelatin and collagen hemostatic agent according to the surgeon's habits; ③ Failure to timely count surgical cotton strips, suture needles and other items before closing the cavity; ④ Failure to prepare hemostatic cotton strips, sponge strips and other items according to the surgeon's surgical habits;

[0128] 9. Return of the bone flap: (1) Preparation of items; bone flap, inlet holder, two inlet holes; operation evaluation: ① Failure to pay attention to the progress of the operation in a timely manner and effectively transfer and recover the bone flap, inlet holder, two inlet holes and other instruments and items; ② Failure to clean the blood stains on the periosteum of the bone flap in a timely manner; ③ Failure to prepare the inlet flap in a timely manner, check the number of inlet flaps and nails, and fix the titanium nail area; ④ Failure to keep the inlet holder upright when transferring the titanium nail to the surgical area and to prevent the nail from falling during the transfer process; ⑤ Failure to replace the medium-sized suction device when returning the bone flap;

[0129] 10. Drainage tube placement: (1) Preparation of items: rongeurs, drainage bottles, sterile cotton balls, scissors, toothed forceps, sharp knives, small curved vascular forceps, muscle sutures; Operation evaluation: ① Failure to pay attention to the progress of the operation in a timely manner and effectively transfer bipolar, suction device, scissors, sharp knives, small curved forceps, muscle sutures and other instruments and items; ② Failure to timely standardize the cutting of drainage tubes and timely check and confirm the side holes; ③ Failure to skillfully and standardize the transfer of corresponding instruments and items according to the steps of drainage surgery and pay attention to the storage of sharps; ④ Failure to prepare appropriate drainage tubes and tube placement methods according to the doctor's surgical habits;

[0130] 11. Suturing of the galea aponeurotica, subcutaneous tissue and skin: (1) Preparation of items; bipolar electrocoagulation, muscle sutures, gauze with teeth forceps; evaluation of operation; ① Failure to pay timely attention to the progress of the operation and effectively deliver bipolar, suction device, muscle sutures, gauze and other instruments and items; ② Failure to timely count the instruments and items after closing the body cavity; ③ Failure to timely flush the surgical area with saline; ④ Failure to timely disinfect the wound; ⑤ Failure to timely and standardizedly prepare gauze dressings.

[0131] In a specific implementation, targeted training modules are provided for the risks arising from each stage of the operation, such as emergency treatment plans for arterial and venous bleeding. Such emergency events and treatments are summarized based on emergency events that routinely occur to doctors.

[0132] The purpose of the embodiment of the present invention is to enable novice nurses to handle the emergency situations of surgery relatively calmly, skillfully, promptly and accurately when simulating the emergency situations of surgery. The emergency events that may occur during surgery include the treatment of arterial and venous bleeding, the treatment of frontal sinus and mastoid air cell opening, the treatment of venous sinus bleeding, and the emergency treatment of the loss of suture needles and other items during surgery, including:

[0133] Obtain the user's completion time for the targeted practice module, preset the standard duration of each operation according to the complexity of the operation, characterize the user's proficiency through the actual duration of the user completing each operation, obtain the user's average proficiency in the targeted practice module based on the completion time of the targeted practice module, and use the average proficiency as the proficiency benchmark to obtain the operation steps in the targeted practice module where the user's proficiency is lower than the proficiency benchmark; evaluate the user's targeted practice module results through the user's completion time information, the number of times marked, the number of operation steps with proficiency lower than the proficiency benchmark, and the proportion of operation steps to the user's completion time information.

[0134] Furthermore, formulating a training plan according to the training results of the user includes:

[0135] Obtain the operation of training surgical instruments and user training results, and vectorize the operations;

[0136] Obtain information on the number of times each operation is marked and the number of times the proficiency is determined to be lower than the proficiency benchmark from the user training results, and generate an attention score for each operation based on the number information;

[0137] Generate an attention-based vectorized representation of each operation in the user training result through the attention score and the vectorized representation of the operation;

[0138] Train and build a training plan recommendation model, input the attention-based vectorized representation of each operation in the user training results into the training plan recommendation model, and generate a recommended training plan.

[0139] In a specific implementation, it also includes: constructing a database, storing the training plans of historical users and digital twins and interaction data related to various types of training content, and storing the basic data of historical users in a preset database in combination with the actual training learning status and the corresponding training plans; extracting learning features based on the training results in the user's current training operation, and obtaining historical user data whose similarity meets the preset similarity threshold range based on the learning features; obtaining the training plans of the historical user data, and sorting them in reverse order according to the similarity, and obtaining the training plan with the highest similarity as the training plan for the user's current preset time period; sending the training plan for the current preset time period to the user in a preset manner, obtaining the user's feedback information on the training plan based on his or her own learning situation, and evaluating the user's fitness for the training plan based on the feedback information; when the fitness is less than the preset fitness threshold, obtaining the training plan with the second highest similarity based on the similarity sorting to replace the training plan for the current preset time period.

[0140] Furthermore, the training and building of the training program recommendation model includes:

[0141] Obtaining the operation of training surgical instruments and the user training results of users, as well as the vectorized representation of each operation in the user training results based on the attention mechanism, performing data preprocessing, and obtaining a training set;

[0142] A training plan recommendation model is constructed based on a neural network with an attention mechanism, and the training is initialized using a training set. A classification model is constructed using a multilayer perceptron with two fully connected layers to recommend training plans. The relu function is selected as the activation function, and the output value is converted into a probability through the sigmoid function. The training effect is updated by coupling the knowledge state to obtain the final recommended training plan.

[0143] In a specific implementation, the output value is converted into a probability by a sigmoid function, and the coupled knowledge state update training effect includes updating the knowledge state:

[0144] After performing a specific training operation, different users’ knowledge states are updated to different degrees. If a user can successfully solve a challenging training operation, then their understanding of the relevant basic concepts should be enhanced accordingly. The update of the user’s knowledge state mainly considers three factors: the user’s prior knowledge state, the user’s operation performance, and the embedded representation of the training operation.

[0145] A learning indicator is used to balance the user's previous knowledge state and the user's current training gains, so as to update their knowledge state and obtain the current state. In this updating process, the learning indicator plays a vital role. It evaluates the user's current training effect based on the user's operation performance and the difficulty of the training operation, and adjusts the update of their knowledge state accordingly. In this way, it can ensure that the user's knowledge state update reflects both the user's previous learning foundation and the user's progress in the current training operation.

[0146]

[0147] Among them, Γ is the gate unit, W1 and β1 are learnable parameters, δ t To practice embedding vectors, σ is the sigmoid activation function;

[0148] h t =(1-Γ)·ω t +Γ·h t-1 , where h t is the user’s knowledge state after the next “time step”. Finally, the hidden layer representation of the user’s knowledge state will be further processed to generate an output state, which represents the user’s current knowledge state.

[0149] μ t =σ(W2·h t +β2);

[0150] Among them, μ t is the user's final knowledge state, W2 and β2 are learnable parameters,

[0151] In order to predict the training effect, the previous output state (representing the user's current knowledge state) is concatenated with the representation of the current training operation. In order to optimize the performance of the model, the model parameters are adjusted by the gradient descent optimization method, with the goal of minimizing the cross entropy loss. This loss function is used to quantify the difference between the training effect predicted by the model and the actual user training effect label.

[0152] in, It is the predicted training effect, which will be compared with the actual training effect in the next step to calculate the pros and cons. t-1 is the user’s knowledge state at the previous “time step”, q t is the training operation; the loss function is

[0153] In a specific implementation, it also includes: setting training stages or modules for different training stages according to the knowledge status update results, so that during the training of instrument nurses, the content of a certain stage can be strengthened according to the learning progress to consolidate the learning effect.

[0154] In a specific embodiment, a cranial surgery training system based on digital twins, such as Figure 2 As shown, including:

[0155] Collection module: used to collect physical data of training surgical instruments;

[0156] 3D reconstruction module: used to generate point cloud data of training surgical instruments based on entity data, and to perform 3D reconstruction on the point cloud data to generate a 3D reconstruction model of the training surgical instruments;

[0157] Interaction module: used to establish a digital twin model based on the 3D reconstruction model of the training surgical instrument, and to build a virtual interaction scene based on the digital twin model;

[0158] Recognition module: used to extract key frames from interactive images of virtual interactive scenes, identify user operations on key frame images, and obtain user operation information during the training process;

[0159] Evaluation module: used to evaluate the user's training results based on the operation information;

[0160] Development module: used to develop training plans based on the user's training results.

[0161] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0162] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A cranial surgery training method based on digital twins, characterized in that: include: Collecting entity data of the training surgical instrument, generating point cloud data of the training surgical instrument according to the entity data, and performing three-dimensional reconstruction on the point cloud data to generate a three-dimensional reconstruction model of the training surgical instrument; A digital twin model is established based on the three-dimensional reconstruction model of the training surgical instrument, and a virtual interactive scene is constructed based on the digital twin model; Extract key frames from interactive images of virtual interactive scenes, identify user operations on key frame images, obtain user operation information during training, and evaluate user training results based on the operation information; Develop training plans based on users' training results.

2. According to claim 1, a cranial surgery training method based on digital twins is characterized in that: The three-dimensional reconstruction of the point cloud data to generate a three-dimensional reconstruction model of the training surgical instrument includes: Acquire multi-angle image information of the training surgical instrument, acquire point cloud data based on the multi-angle image information, pre-process the point cloud data, and acquire the size and assembly information of the training surgical instrument; Perform 3D reconstruction through point cloud data, align and connect the point cloud data according to size and assembly information, and generate a 3D reconstruction model of the training surgical instrument; Assign the material properties, electromechanical object properties and motion constraints of the training surgical instrument to the 3D reconstruction model to build a digital twin model of the training surgical instrument; Motion-match the digital twin model of the training surgical instrument to the training surgical instrument.

3. According to claim 1, a cranial surgery training method based on digital twins is characterized in that: The obtaining of the user's operation information during the training process includes: Establish the connection and mapping relationship between the digital twin model and the training surgical instrument, and perform data transmission of the digital twin model; A virtual interactive scene is constructed through the digital twin model of the training surgical instrument, and relevant training teaching videos are preset in the virtual interactive scene to guide the user's operation; Key frame extraction is performed on the interactive image of the virtual interactive scene obtained by data transmission, and the user's operation is identified on the key frame image.

4. According to claim 3, a cranial surgery training method based on digital twins is characterized in that: The operation of identifying the user includes: The user's operation is divided into the user's hand movements and the training surgical instrument selected by the user; Matching the image features of the key frame image with the image features of the pre-stored hand movements, and taking the type of the hand movement corresponding to the highest similarity as the type of the user's hand movement; Matching the image features of the training surgical instrument in the key frame image with the image features of the pre-stored training surgical instrument, and taking the type of the training surgical instrument corresponding to the highest similarity as the type of the training surgical instrument selected by the user; The key frame image features are matched with the pre-stored operation image features, and the type of operation corresponding to the highest similarity is identified as the type of user operation.

5. According to claim 4, a cranial surgery training method based on digital twins is characterized in that: Also includes: When the user's operation deviates from the operation in the teaching video, the operation is marked; At the same time, the corresponding motion variables are generated according to the operation of the digital twin model, and the motion extreme values ​​are generated by training the motion connection and motion constraints of each component in the surgical instrument. When the motion variable is greater than the motion extreme value, it is marked.

6. According to claim 1, a cranial surgery training method based on digital twins is characterized in that: The training result of evaluating the user according to the operation information includes: Obtain the user's training completion time, preset the standard duration of each operation according to the complexity of the operation, characterize the user's proficiency through the actual duration of each operation completed by the user, calculate the user's average proficiency during training based on the training completion time, and use the average proficiency as the proficiency benchmark to obtain the operation steps in which the user's proficiency during training is lower than the proficiency benchmark; The user training results are evaluated through the time information of the user's completion, the number of times marked, the number of operation steps with proficiency lower than the proficiency benchmark, and the proportion of the operation steps to the time information of the user's completion.

7. According to claim 1, a cranial surgery training method based on digital twins is characterized in that: The training program formulated according to the user's training results includes: Obtain the operation of training surgical instruments and user training results, and vectorize the operations; Obtain information on the number of times each operation is marked and the number of times the proficiency is determined to be lower than the proficiency benchmark from the user training results, and generate an attention score for each operation based on the number information; Generate an attention-based vectorized representation of each operation in the user training result through the attention score and the vectorized representation of the operation; Train and build a training plan recommendation model, input the attention-based vectorized representation of each operation in the user training results into the training plan recommendation model, and generate a recommended training plan.

8. According to claim 7, a cranial surgery training method based on digital twins is characterized in that: The training and construction of the training program recommendation model includes: Obtaining the operation of training surgical instruments and the user training results of users, as well as the vectorized representation of each operation in the user training results based on the attention mechanism, performing data preprocessing, and obtaining a training set; A training plan recommendation model is constructed based on a neural network with an attention mechanism, and the training is initialized using a training set. A classification model is constructed using a multilayer perceptron with two fully connected layers to recommend training plans. The relu function is selected as the activation function, and the output value is converted into a probability through the sigmoid function. The training effect is updated by coupling the knowledge state to obtain the final recommended training plan.

9. A cranial surgery training system based on digital twins, characterized in that: include: Collection module: used to collect physical data of training surgical instruments; 3D reconstruction module: used to generate point cloud data of training surgical instruments based on entity data, and to perform 3D reconstruction on the point cloud data to generate a 3D reconstruction model of the training surgical instruments; Interaction module: used to establish a digital twin model based on the 3D reconstruction model of the training surgical instrument, and to build a virtual interaction scene based on the digital twin model; Recognition module: used to extract key frames from interactive images of virtual interactive scenes, identify user operations on key frame images, and obtain user operation information during the training process; Evaluation module: used to evaluate the user's training results based on the operation information; Development module: used to develop training plans based on the user's training results.

Images