A pediatric endotracheal intubation training system and method based on augmented reality

Abstract

The present invention relates to a pediatric endotracheal intubation training system and method based on augmented reality, including an augmented reality module, an endotracheal intubation entity model module, a medical behavior recognition and evaluation module and a voice interaction module; the augmented reality module includes augmented reality glasses, and the augmented reality module is used to provide users with augmented reality images and audio; the endotracheal intubation entity model module is used to provide users with a physical model that simulates the completion of endotracheal intubation operations; the medical behavior recognition and evaluation module is used to identify the user's behavior during the endotracheal intubation process and give real-time evaluation; the voice interaction module is used to issue instructions or replies to the user, while simulating the sound emitted when the endotracheal intubation operation is performed in the clinic. The present invention integrates technologies such as augmented reality, voice interaction, and deep learning to make the training method more professional and efficient. Compared with the prior art, the present invention has the advantages of low cost, saving teaching time, and high teaching efficiency.

Description

Technical Field

[0001] The present invention relates to the field of medical technology, and in particular to a pediatric endotracheal intubation training system and method based on augmented reality. Background Art

[0002] Tracheal intubation is an important and common emergency rescue technique in clinical practice. At present, the intubation teaching usually begins with the instructor using PPT to explain the theory, then demonstrating the operation on a model, and the students imitate and practice. However, practice has shown that under this teaching mode, students are not very interested because the teaching content is boring, and students have few opportunities for independent learning and exploration, and cannot master all relevant knowledge in a short period of time. Therefore, it is difficult to quickly master intubation skills through this teaching method; at the same time, due to the difference between the teaching and training scenarios and the actual diagnosis and treatment environment, students do not understand the actual situation that may occur in the clinic. Therefore, even if they pass the relevant tracheal intubation operation assessment, it is difficult to apply the technology in the clinic as soon as possible.

[0003] At present, the mainstream teaching method of tracheal intubation is that the instructor first explains the theory, then demonstrates the operation on the physical model, and finally the students practice and are assessed by the instructor. Some teaching units use video laryngoscopes instead of traditional laryngoscopes, so that students can see the various anatomical structures of the oropharynx during the instructor's demonstration and understand the dynamic process of laryngoscope insertion into the mouth and pharynx; some physical models are equipped with photoelectric gates in the larynx or pressure sensors in the teeth to alarm when students mistakenly insert into the esophagus or compress the teeth. Although the above methods have improved some shortcomings of current tracheal intubation teaching, they have not fundamentally changed the traditional teaching mode of teacher explanation, demonstration, and assessment, and the intelligent and professional level of tracheal intubation teaching has not been significantly improved.

[0004] Some units have introduced the PBL (problem-based learning) model in tracheal intubation teaching, requiring students to read relevant case materials before tracheal intubation teaching and find out various problems that may arise during tracheal intubation in pediatric clinics. Studies have shown that this method enhances students' learning interest and learning ability, but this teaching model still remains in theory, and students are still unable to combine theory with practice in this model, and the authenticity and efficiency of training have not been improved.

[0005] Under the current endotracheal intubation training model, relevant statistics from a certain hospital show that only 35% of junior residents have mastered endotracheal intubation techniques, and 18.5% of hospitals rely entirely on anesthesiologists for endotracheal intubation operations.

[0006] At the same time, pediatric endotracheal intubation lacks a more specific and refined teaching model. In practice, endotracheal intubation in children is very different from that in adults. The airway of adults is relatively fixed, while the airway of children is more complicated, with many congenital malformations. The larynx of children is located more forward and is not easy to expose. Children are sensitive to sedatives and muscle relaxants, and they are rarely used before intubation. Therefore, resistance will occur in clinical practice that is not encountered in model training.

[0007] In addition, traditional tracheal intubation training only provides operational training for related models, without combining relevant cases or related emergencies that may be encountered in clinical practice. At the same time, although the mainstream physical models are equipped with some sensors, such as those for detecting tooth compression and misinsertion of the esophagus, they all respond in the form of alarms, with a low level of intelligence. In actual clinical operations, children will not tell doctors whether there are any misoperations. Therefore, this training method cannot truly improve doctors' ability to combine theory with practice. Moreover, in the operational assessment of doctors in regular training, experienced teaching teachers are often required to evaluate students one by one, which greatly wastes doctors' time and reduces evaluation efficiency.

[0008] Therefore, it is necessary to design a new pediatric endotracheal intubation training system to address the above problems, improve doctors' learning efficiency, free teachers from tedious operation evaluations, make pediatric endotracheal intubation training more professional and refined, make endotracheal intubation training more realistic, more simulate clinical practice, and improve doctors' ability to combine theory with practice. Summary of the invention

[0009] The purpose of the present invention is to provide a pediatric endotracheal intubation training system and method based on augmented reality in order to overcome the defects of the above-mentioned prior art. The present invention can automatically assess the physician's operation, making pediatric endotracheal intubation training more professional and refined; at the same time, it improves the intelligence level of the training system, making intubation training more realistic and closer to the clinic, so that physicians can better combine theory with practice, thereby effectively improving the professional level of pediatric physicians and providing a new technical solution for ensuring the quality of medical emergency care.

[0010] The purpose of the present invention can be achieved by the following technical solutions:

[0011] A pediatric endotracheal intubation training system based on augmented reality, including an augmented reality module, an endotracheal intubation physical model module, a medical behavior recognition and evaluation module, and a voice interaction module;

[0012] The medical behavior recognition and evaluation module is connected to the augmented reality module;

[0013] The augmented reality module includes augmented reality glasses, and the augmented reality module is used to provide augmented reality images and audio to the user;

[0014] The endotracheal intubation entity model module is used to provide a user with an entity model that simulates the endotracheal intubation operation;

[0015] The medical behavior recognition and evaluation module is used to recognize the user's behavior during tracheal intubation and provide real-time evaluation;

[0016] The voice interaction module is used to issue instructions or responses to the user, and at the same time simulate the sound produced when performing tracheal intubation operations in clinical practice.

[0017] Furthermore, the augmented reality glasses include a memory unit, and the memory unit is used to store part of the intubation operation video.

[0018] Furthermore, the endotracheal intubation physical model module includes a child physical model, a video laryngoscope and an endotracheal intubation device.

[0019] Furthermore, the voice interaction module includes a voice device and voice recognition software.

[0020] Furthermore, the medical behavior recognition and evaluation module includes a camera, an information collection device and a controller.

[0021] Furthermore, the information acquisition equipment includes an IMU, a photoelectric gate and a pressure sensor.

[0022] Furthermore, a behavior recognition and evaluation technical solution based on deep learning is deployed in the controller of the medical behavior recognition and evaluation module, which is used to identify the state of the operator during the tracheal intubation process. The solution includes the following steps:

[0023] Use the AlphaPose algorithm to detect the operator's posture, extract skeleton key points, and perform behavior recognition and analysis based on skeleton key points;

[0024] Eliminate redundant pose detection using parameterized pose non-maximum suppression;

[0025] The MLP multi-layer perceptron recognizes the operator's medical behavior based on the detected skeleton key points.

[0026] Furthermore, there are two IMUs, which are worn on both hands of the user. The controller collects IMU data to perform fine-grained medical behavior recognition, including the following steps:

[0027] Input the IMU data into two LSTM neural networks respectively;

[0028] The IMU data is subjected to feature extraction by two LSTM layers and then feature fusion;

[0029] The fused features are passed through the softmax layer to output fine-grained medical behavior recognition results.

[0030] A pediatric endotracheal intubation training method based on augmented reality is applied to a pediatric endotracheal intubation training system based on augmented reality as described above, including a learning mode and a practice mode, wherein the learning mode includes the following steps:

[0031] Recording a teaching video of the teacher performing the intubation operation, the teaching video includes the teacher's operation process and the video laryngoscope insertion process recorded by the camera in the medical behavior recognition and evaluation module, and the video of the internal structure of the larynx recorded by the video recording device set on the video laryngoscope;

[0032] Storing the teaching video in an augmented reality module;

[0033] The operator uses the augmented reality glasses in the augmented reality module to learn. The augmented reality glasses show the operator the teaching video and the video recorded by the video laryngoscope. At the same time, the voice interaction module provides real-time operation guidance until the operator completes the learning.

[0034] Furthermore, the practice mode includes the following steps:

[0035] The augmented reality glasses show the operator the background and basic symptoms of the virtual patient, and present emergencies in a timely manner based on the specific case;

[0036] The operator uses the voice interaction system to answer questions about the virtual patient's condition and the medical procedures that need to be performed;

[0037] The operator performs tracheal intubation on the child physical model. During the operation, the medical behavior recognition and evaluation module will record the operator's actions and conduct an evaluation;

[0038] When the operator makes an error in operation, the medical behavior recognition and evaluation module calculates the possible consequences of the error operation, and maps the consequences onto the child entity model through the augmented reality module;

[0039] After the operator finishes the operation, the medical behavior recognition and evaluation module evaluates the operator's entire operation process and outputs the evaluation results.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. Aiming at the current uneven distribution of medical resources in my country, especially the lack of accessibility to medical and health services for children, the present invention proposes a pediatric endotracheal intubation training system based on augmented reality and multiple sensors. The system aims to address the shortcomings of traditional pediatric endotracheal intubation training, such as single means, low intelligence level, large gap between theory and practice, and lack of specialization and refinement. It integrates technologies such as augmented reality, voice interaction, and deep learning to make the training method more professional and efficient.

[0042] 2. In the learning mode of the present invention, beginners can quickly master the basic operations of tracheal intubation step by step by combining the operation teaching video in the augmented reality glasses and the guidance of the voice interaction module, which reduces the students' learning time and number of attempts and enhances their memory of basic knowledge.

[0043] 3. In the practice mode of the present invention, technologies such as augmented reality, voice interaction, and deep learning are all used in the teaching process of pediatric endotracheal intubation. Among them, the teaching is problem-oriented (PBL). The augmented reality glasses first use specific cases as the background to examine the pediatric students' mastery of knowledge of different diseases. Subsequently, the students began the endotracheal intubation operation. During the operation, the behavior recognition and evaluation system constantly collected the students' movements, and combined the students' operations with real cases, and various types of sudden symptoms were displayed on the virtual patient model through augmented reality and voice interaction modules, and the blood pressure and ECG information of the virtual children were displayed in time for students to analyze, combining theory with practice, greatly improving the authenticity of the endotracheal intubation operation, improving students' thinking and analysis abilities, improving students' enthusiasm for learning, deepening students' understanding of various clinical problems, and improving students' ability to respond to emergencies in pediatric endotracheal intubation clinics and clinical hands-on ability, making pediatric endotracheal intubation training more professional, detailed, and real than ordinary endotracheal intubation training. Finally, the medical behavior recognition and evaluation module gives the evaluation results of the student's entire operation process, and uses deep learning methods to replace the traditional evaluation of teachers, which greatly saves human resources, improves evaluation efficiency, and saves teaching time. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic diagram of the system structure of the present invention;

[0045] Figure 2 A flowchart of the learning mode steps;

[0046] Figure 3 It is a flowchart of the steps of the practice mode;

[0047] Figure 4 This is a flowchart of the vision-based behavior recognition and evaluation technology solution;

[0048] Figure 5 This is a flow chart of the IMU-based fine-grained medical behavior recognition method. DETAILED DESCRIPTION

[0049] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0050] like Figure 1 As shown, the present invention proposes a pediatric endotracheal intubation training system based on augmented reality, including an augmented reality module, an endotracheal intubation entity model module, a medical behavior recognition and evaluation module, and a voice interaction module;

[0051] The medical behavior recognition and assessment module is connected to the augmented reality module;

[0052] The augmented reality module includes augmented reality glasses, which are used to provide users with augmented reality images and audio, including text, images, audio information, etc. The augmented reality glasses include a memory unit, which is used to store part of the intubation operation video.

[0053] The endotracheal intubation physical model module includes a child physical model, a video laryngoscope and an endotracheal intubation device. The endotracheal intubation physical model module is used to provide users with a physical model for simulating the endotracheal intubation operation; in this embodiment, the child physical model adopts a virtual child physical model with accurate upper respiratory tract anatomical structure, realistic airway, realistic anthropomorphic morphology and obvious body surface marks.

[0054] The medical behavior recognition and evaluation module includes a camera, an information acquisition device and a controller. The information acquisition device includes an IMU (Inertial measurement unit), a photoelectric gate and a pressure sensor. The photoelectric gate is used to detect the insertion status of the intubation tube; the pressure sensor is used to detect whether the video laryngoscope exerts pressure on the teeth of the child's physical model. The medical behavior recognition and evaluation module is used to identify the user's behavior during the tracheal intubation process and provide real-time evaluation.

[0055] The voice interaction module includes voice equipment and voice recognition software. The voice interaction module is used to issue instructions or responses to the user, and simulate the sound produced when performing tracheal intubation operations in clinical practice.

[0056] The controller of the medical behavior recognition and evaluation module is deployed with a behavior recognition and evaluation technology solution based on deep learning, which is used to identify the medical behavior of the operator during tracheal intubation. The solution includes the following steps:

[0057] (1) Use the AlphaPose algorithm to detect the operator's posture, extract skeleton key points, and perform behavior recognition and analysis based on the skeleton key points;

[0058] (2) Use parameterized posture non-maximum suppression to eliminate redundant posture detection;

[0059] (3) The operator's medical behavior is evaluated by MLP (Multilayer Perceptron) based on the detected skeleton key points.

[0060] In this embodiment, two IMUs are provided, and the IMUs are used to be worn on both hands of the user. The controller collects IMU data to perform fine-grained medical behavior recognition, including the following steps:

[0061] (1) Input the IMU data into two LSTM (long short-term memory) neural networks respectively;

[0062] (2) The IMU data is subjected to feature extraction by two LSTM layers and then feature fusion is performed;

[0063] (3) The fused features are passed through the softmax layer to output fine-grained medical behavior recognition results.

[0064] This embodiment also provides a pediatric endotracheal intubation training method based on augmented reality, which is applied to a pediatric endotracheal intubation training system based on augmented reality as described above, including a learning mode and a practice mode. The learning mode is used for beginners to gradually learn endotracheal intubation operations; the practice mode is used for users who are already familiar with endotracheal intubation operations to perform more realistic pediatric endotracheal intubation operations based on the PBL method and combined with actual cases.

[0065] like Figure 2 As shown, the learning mode includes the following steps:

[0066] Recording a teaching video of the teacher performing the intubation operation, the teaching video includes the teacher's operation process and the video laryngoscope insertion process recorded by the camera in the medical behavior recognition and evaluation module, and the video of the internal structure of the larynx recorded by the video recording device set on the video laryngoscope;

[0067] Storing the teaching video in an augmented reality module;

[0068] The operator uses the augmented reality glasses in the augmented reality module to learn. The augmented reality glasses show the operator the teaching video and the video recorded by the video laryngoscope. At the same time, the voice interaction module provides real-time operation guidance until the operator completes the learning.

[0069] Practice mode consists of the following steps:

[0070] The augmented reality glasses show the operator the background and basic symptoms of the virtual patient, and present emergencies in a timely manner based on the specific case;

[0071] The operator uses the voice interaction system to answer questions about the virtual patient's condition and the medical procedures that need to be performed;

[0072] The operator performs tracheal intubation on the child physical model. During the operation, the medical behavior recognition and evaluation module will record the operator's actions and conduct an evaluation;

[0073] When the operator makes an error in operation, the medical behavior recognition and evaluation module calculates the possible consequences of the error operation, and maps the consequences onto the child entity model through the augmented reality module;

[0074] After the operator finishes the operation, the medical behavior recognition and evaluation module evaluates the operator's entire operation process and outputs the evaluation results.

[0075] In this embodiment, in view of the common mistakes in intubation, the medical behavior recognition and evaluation module will present the emergency adverse reactions of the virtual patient by identifying the wrong operation of the physician, and present some sudden symptoms in combination with the specific case, mainly including the following points:

[0076] 1. When the teeth are pressed too hard during the operation, the operator will see slight bleeding on the virtual patient's teeth in the augmented reality glasses.

[0077] 2. When the catheter is not properly selected or the intubation time is too long, the operator will see symptoms of laryngeal edema in the virtual patient captured by the video laryngoscope in the augmented reality glasses.

[0078] 3. When the operator moves too violently, the operator will hear the voice interaction module prompt that the patient's tongue, trachea, vocal cords or esophagus are damaged.

[0079] 4. If the child is already in a state of hypoxia and exhaustion, and is further deprived of oxygen due to struggling after intubation, the operator will see the virtual child exhibit symptoms such as cyanosis in the augmented reality glasses.

[0080] 5. If the endotracheal intubation causes excessive stimulation to the child, the operator will see symptoms such as severe choking and shortness of breath in the child through the augmented reality glasses.

[0081] 6. If the intubation operation stimulates the vagus nerve in the throat, the ECG signal of the child displayed in the augmented reality glasses will show symptoms of bradycardia.

[0082] 7. If intubation causes excessive excitement of the child's autonomic nervous system, the ECG signal of the child displayed in the augmented reality glasses will show symptoms such as bradycardia, arrhythmia, or even cardiac arrest, tachycardia, ventricular premature beats or ventricular fibrillation.

[0083] 8. If the operator mistakenly inserts the catheter into the esophagus, the operator will see the child's abdomen bulging in the augmented reality glasses, and will hear the sound of upper abdominal air intake and the child's voice emitted by the voice interaction module.

[0084] 9. If the child vomits without gastric tube suction, laryngoscope stimulation or cardiac compression before intubation, the operator will see symptoms of dry heaves or vomiting in the child through the augmented reality glasses.

[0085] In this embodiment, the controller of the medical behavior recognition and evaluation module collects case data and clinical operation data. Based on such data, the controller can randomly output relevant sudden symptoms that may be caused by tracheal intubation according to the specific case, and send the sudden symptoms to the augmented reality module for display.

[0086] During the doctor's operation, the medical behavior recognition and evaluation module of this system will recognize the patient's actions in real time, and generate corresponding text, video or audio according to the patient's operation and the case of the virtual child for the operator to judge, analyze and process. The operator needs to respond accordingly according to the symptoms, such as correcting the operation or even stopping the intubation; for symptoms that cannot be dealt with by existing tools, the operator will verbally state the solution that should be taken, and the voice interaction module will receive and judge whether it is correct or not.

[0087] When the patient's complete operation process is completed, the behavior recognition and evaluation module completes the evaluation of the operator's operation and displays the results in text on the augmented reality glasses to inform the operator.

[0088] The medical behavior recognition and evaluation module in this embodiment is composed of multiple sensors and multiple neural networks. The sensors include cameras, IMUs, photoelectric gates, and pressure sensors; two cameras are installed on the side and above the operating table, two IMUs are worn on the operator's left and right hands, and photoelectric gates and pressure sensors are installed on the intubation physical model, which are used to detect whether the catheter is inserted into the trachea and whether the laryngoscope compresses the teeth. For the evaluation of pediatric endotracheal intubation operation specifications, the present invention designs a behavior recognition and evaluation technical solution based on deep learning.

[0089] For most medical behaviors, the solution uses the AlphaPose algorithm to detect key points of the human skeleton and perform behavior recognition and analysis based on the key points of the human skeleton. Figure 4 As shown in the figure, the system adopts a top-down approach. First, the human body detection frame in the environment is detected, and the key points of the human skeleton are identified based on the features in the detection frame. Then, parameterized posture non-maximum suppression is used to efficiently eliminate redundant posture detections. Finally, the MLP multi-layer perceptron evaluates human body movements based on the detected key points of the human skeleton.

[0090] At the same time, there are many fine-grained actions in this scenario. These actions are highly correlated and have high inter-class similarity, which makes them difficult to identify using video-based methods. Therefore, the present invention selects to collect IMU data worn on the operator's hands for fine-grained medical behavior recognition. The fine-grained medical behavior recognition method used in the present invention is as follows: Figure 5 As shown in the figure, we input the two IMU data into two LSTM neural networks respectively, and each of them is subjected to feature extraction by two LSTM layers, and then feature fusion is performed, and then the results are output by the softmax layer. The above method is used to identify the user's medical behavior and give a final evaluation.

[0091] The system addresses the shortcomings of traditional pediatric endotracheal intubation training, such as single means, low intelligence level, large gap between theory and practice, and lack of specialization and refinement. It cleverly integrates cutting-edge technologies such as augmented reality, voice interaction, and deep learning, so that every student learning pediatric endotracheal intubation operations can use more professional and efficient training technology.

[0092] In learning mode:

[0093] Beginners can quickly master the basic operations of tracheal intubation step by step by combining the operations of the teaching teacher seen in the augmented reality glasses and the guidance of the voice interaction module, which reduces students' learning time and number of attempts and enhances their memory of basic knowledge.

[0094] In practice mode:

[0095] Cutting-edge technologies such as augmented reality, voice interaction, and deep learning are all used in the teaching of pediatric endotracheal intubation.

[0096] Among them, the teaching is problem-oriented. The augmented reality glasses first use specific cases as the background to examine pediatric students' knowledge of different diseases.

[0097] Afterwards, the students began the tracheal intubation operation. During the operation, the behavior recognition and evaluation system collected students' movements at all times, and combined with students' operations and real cases, various types of sudden symptoms were displayed on the virtual patient model through augmented reality and voice interaction modules, and the blood pressure and ECG information of the virtual child were displayed in a timely manner for students to analyze. The combination of theory and practice greatly improved the authenticity of the tracheal intubation operation, enhanced students' thinking and analytical abilities, increased students' enthusiasm for learning, deepened students' understanding of various clinical problems, and enhanced students' ability to respond to emergencies in pediatric tracheal intubation clinics and clinical hands-on ability, making pediatric tracheal intubation training more professional, detailed, and realistic than ordinary tracheal intubation training.

[0098] Finally, the medical behavior recognition and evaluation module gives the evaluation results of the student's entire operation process, and uses deep learning methods to replace the traditional evaluation of teachers, which greatly saves human resources, improves evaluation efficiency, and saves teachers' time.

[0099] In order to better explain the implementation mode of the present invention, a virtual child suffering from NRDS (neonatal respiratory distress syndrome) will be taken as an example to explain the workflow of the pediatric endotracheal intubation training system.

[0100] First, the augmented reality glasses displayed the basic information of the virtual patient: a 6-hour-old newborn who needed emergency treatment due to "progressive dyspnea for 5 hours". The child had dyspnea and rapid breathing (more than 60 times / minute) one hour after birth.

[0101] At the same time, the students saw through the augmented reality glasses that the child had symptoms of rapid breathing, expiratory groaning, and cyanosis of the face.

[0102] Then, the voice interaction module asks the students a question: "Please make a preliminary judgment on the child's condition." The students need to answer their inference about the child's condition and indicate which routine examinations need to be conducted.

[0103] In the next step, the system will show students the test results of the virtual child, such as blood routine, blood gas analysis, chest X-ray, etc., in augmented reality glasses. The voice interaction module requires students to analyze the test results and make further inferences about the condition and the operations that need to be performed.

[0104] The student completes the analysis of the examination results and points out that endotracheal intubation and mechanical ventilation are required. The voice interaction module prompts the patient to perform endotracheal intubation.

[0105] Then the students began to perform the tracheal intubation operation, and the medical behavior recognition and evaluation system began to work. The students first completed the pre-intubation preparations, then stood at the head of the bed and verbally informed the system that the intubation had begun. The students made the virtual child lie on his back, put a small pillow on his shoulders, and tilted his head back so that the mouth, pharynx, and larynx were in the same straight line. Check the mouth, open the sterile tray, and wear gloves. Test whether the air bag is leaking, insert the catheter core, and lubricate the catheter and the front end of the lens with paraffin gauze. The system detected that the catheter core was selected incorrectly or not lubricated. After the students intubated, they saw that the virtual child had severe choking, coughing, and breath holding symptoms. After completing the previous step, use the thumb, index, and middle fingers of the right hand to separate the upper and lower lips, hold the laryngoscope with the left hand and insert it into the mouth along the right corner of the mouth, push the tongue to the left with the side of the lens, move the laryngoscope to the middle position, and use the left arm to lift it up to expose the mouth. At this time, the system detects that the teeth are upturned with the teeth as the fulcrum, and the students see the virtual child's teeth bleeding slightly in the augmented reality glasses. After the balloon of the endotracheal tube passes the glottis, pull out the catheter core and continue to insert it to the required depth. At this time, the ECG signal of the virtual child is displayed in the augmented reality glasses. After analysis, the students conclude that the child has bradycardia and the intubation operation should be stopped immediately. They verbally answer what emergency operations should be performed. After the virtual child's heartbeat returns to normal, the students continue the intubation operation. Insert the dental pad, remove the laryngoscope, connect the endotracheal tube with a simple respirator (the assistant assists in squeezing the balloon), auscultate both lungs, and confirm that the catheter is in the trachea; at this time, the students see the child's abdomen bulge in the augmented reality glasses, and hear the voice interaction module

[0106] The sound of air intake in the upper abdomen and the child's voice indicate that the esophagus was inserted by mistake. After correction, the catheter was successfully inserted into the trachea, the catheter and the dental pad were fixed, and about 6 to 8 ml of air was injected into the balloon with a syringe to close the airway. After completion, the tube was removed and the key points of the removal were orally stated (clearing secretions in the oral cavity, pharyngeal cavity and catheter; releasing the balloon; removing the tube while suctioning and guiding, and cleaning the patient's lips); raising hands to indicate the end of the operation.

[0107] Finally, students will see the medical behavior assessment results and wrong behavior records of the entire process on augmented reality glasses, and the pediatric endotracheal intubation training is completed.

[0108] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A pediatric endotracheal intubation training system based on augmented reality, characterized in that: It includes augmented reality module, endotracheal intubation physical model module, medical behavior recognition and evaluation module and voice interaction module; The medical behavior recognition and evaluation module is connected to the augmented reality module; the medical behavior recognition and evaluation module includes a camera, an information acquisition device and a controller; the information acquisition device includes an IMU, a photoelectric gate and a pressure sensor; there are two IMUs, which are worn on both hands of the user. The controller collects IMU data to perform fine-grained medical behavior recognition, including the following steps: Input the IMU data into two LSTM neural networks respectively; The IMU data is subjected to feature extraction by two LSTM layers and then feature fusion; The fused features are passed through the softmax layer to output fine-grained medical behavior recognition results; The augmented reality module includes augmented reality glasses, and the augmented reality module is used to provide augmented reality images and audio to the user; The endotracheal intubation entity model module is used to provide a user with an entity model that simulates the endotracheal intubation operation; The medical behavior recognition and evaluation module is used to identify the user's behavior during the tracheal intubation process and provide real-time evaluation; the controller of the medical behavior recognition and evaluation module is deployed with a behavior recognition and evaluation technical solution based on deep learning, which is used to identify the state of the operator during the tracheal intubation process. The behavior recognition and evaluation technical solution based on deep learning includes the following steps: Use the AlphaPose algorithm to detect the operator's posture, extract skeleton key points, and perform behavior recognition and analysis based on skeleton key points; Eliminate redundant pose detection using parameterized pose non-maximum suppression; The MLP multi-layer perceptron recognizes the operator's medical behavior based on the detected skeleton key points; The voice interaction module is used to issue instructions or responses to the user, and at the same time simulate the sound produced when performing tracheal intubation operations in clinical practice.

2. The augmented reality-based pediatric endotracheal intubation training system according to claim 1, characterized in that: The augmented reality glasses include a memory unit, which is used to store part of the intubation operation video.

3. The augmented reality-based pediatric endotracheal intubation training system according to claim 1, characterized in that: The endotracheal intubation entity model module comprises a child entity model, a video laryngoscope and an endotracheal intubation device.

4. The augmented reality-based pediatric endotracheal intubation training system according to claim 1, characterized in that: The voice interaction module includes voice equipment and voice recognition software.

5. A pediatric endotracheal intubation training method based on augmented reality, characterized in that: A pediatric endotracheal intubation training system based on augmented reality as described in any one of claims 1 to 4, comprising a learning mode, wherein the learning mode comprises the following steps: Recording a teaching video of the teacher performing the intubation operation, the teaching video includes the teacher's operation process and the video laryngoscope insertion process recorded by the camera in the medical behavior recognition and evaluation module, and the video of the internal structure of the larynx recorded by the video recording device set on the video laryngoscope; Storing the teaching video in an augmented reality module; The operator uses the augmented reality glasses in the augmented reality module to learn. The augmented reality glasses show the operator the teaching video and the video recorded by the video laryngoscope. At the same time, the voice interaction module provides real-time operation guidance until the operator completes the learning.

6. The method for pediatric endotracheal intubation training based on augmented reality according to claim 5, characterized in that: Also included is a practice mode, the practice mode comprising the following steps: The augmented reality glasses show the operator the background and basic symptoms of the virtual patient, and present emergencies in a timely manner based on the specific case; The operator uses the voice interaction system to answer questions about the virtual patient's condition and the medical procedures that need to be performed; The operator performs tracheal intubation on the child physical model. During the operation, the medical behavior recognition and evaluation module will record the operator's actions and conduct an evaluation; When the operator makes an error in operation, the medical behavior recognition and evaluation module calculates the possible consequences of the error operation, and maps the consequences onto the child entity model through the augmented reality module; After the operator finishes the operation, the medical behavior recognition and evaluation module evaluates the operator's entire operation process and outputs the evaluation results.

Images