Airway Management Virtual Reality Training

Abstract

An airway management virtual reality (VR) training system and method is provided that uses a relatively simple, passive physical patient model to train users in performing airway management procedures. The patient model may include an anatomically realistic airway, a movable head, and an openable / closable jaw, along with electromagnetic sensors that measure the movement of parts of the physical model and the movement of tools used to manipulate the model. Parameters of the medical procedure being performed are sensed, tracked, and displayed to provide the trainee with a continuous, detailed, and consistent VR representation of the training situation in a realistic scene, utilizing and modifying a hierarchy of sensors to focus the VR representation on key aspects of the training, and obtaining evaluation of the VR patient's interactivity and trainee's performance.

Description

[Technical field]

[0001] The present invention relates to the field of virtual reality systems for medical training, and more specifically to airway management training systems. [Background technology]

[0002] Prior art airway management training systems utilize mechanically complex patient models that include multiple motors and associated actuators, and complex model structural features to make the patient model as realistic as possible. Summary of the Invention

[0003] The following is a brief summary to provide an initial understanding of the invention. This summary does not necessarily identify key elements or delimit the scope of the invention, but is merely intended to serve as a guide for the description that follows.

[0004] One aspect of the invention provides an airway management training system that includes a physical, typically passive patient model having an airway, a head that can move from side to side, and a jaw that can open and close; a plurality of electromagnetic sensors configured to measure the relative positions of the head, the jaw, and at least one airway management tool with respect to the airway; and a virtual reality (VR) system configured to provide a user or other trainee with a VR representation of a scene, of at least one patient corresponding to the physical patient model, of the at least one airway management tool, and of the trainee's hands operating or handling the at least one airway management tool.

[0005] One aspect of the invention provides a method for airway management training comprising the steps of: training airway management within a virtual reality (VR) environment; augmenting the VR environment with a physical patient model and at least one airway management tool, the physical patient model having an airway, a head that can move left and right, and a jaw that can open and close, the at least one airway management tool having at least one pressure sensor; measuring the relative positions of the head and chin and the at least one airway management tool with respect to the airway using a plurality of electromagnetic sensors associated with the passive patient model; and displaying to the trainee within the VR environment a scene, a patient corresponding to the physical patient model, the at least one airway management tool, and the trainee's hands.

[0006] These additional and / or other aspects and / or advantages of the invention will be set forth in the following detailed description, may be inferred from the detailed description, and / or may be learned by practice of the invention.

[0007] For a better understanding of embodiments of the present invention and to show how the same may be put into practice, reference will now be made, by way of example only, to the accompanying drawings in which like numerals refer to corresponding elements or parts throughout and in which: [Brief description of the drawings]

[0008] [Figure 1A] FIG. 1A is a high level schematic block diagram of an airway management training system according to some embodiments of the present invention. [Figure 1B] FIG. 1B is a high level schematic block diagram of an airway management training system according to some embodiments of the present invention. [Diagram 2] FIG. 2 is a high level schematic diagram of components of an airway management training system according to some embodiments of the present invention. [Diagram 3] FIG. 3 is a high level schematic diagram of components of an airway management training system according to some embodiments of the present invention. [Figure 4]FIG. 4 is a high level schematic diagram of components of an airway management training system according to some embodiments of the present invention. [Figure 5A] FIG. 5A provides an example of a VR representation of a scene, a patient, a tool, and a trainee's hands according to some embodiments of the present invention. [Figure 5B] FIG. 5B provides an example of a VR representation of a scene, a patient, a tool, and a trainee's hand according to some embodiments of the present invention. [Figure 6] FIG. 6 is a high level flow chart illustrating a method for airway management training according to some embodiments of the present invention. [Figure 7] FIG. 7 is a high-level block diagram of an exemplary computing device that may be used in embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] In the following description, various aspects of the present invention will be described. For the purpose of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced without the specific details presented herein. Also, well-known features may be omitted or simplified so as not to obscure the present invention. With particular reference to the drawings, it is emphasized that the details shown are presented by way of example only for the purpose of illustrative discussion of the present invention, and are presented to provide what is believed to be the most useful and easily understood explanation of the principles and conceptual aspects of the present invention. In this regard, no attempt is made to show structural details of the present invention in more detail than is necessary for a fundamental understanding of the present invention, and the description taken together with the drawings will make clear to those skilled in the art how some forms of the present invention can be embodied in practice.

[0010] Before describing at least one embodiment of the invention in detail, it is to be understood that the invention is not limited in its application to the details of construction and the arrangement of components set forth in the following description or illustrated in the drawings. The invention is applicable to other embodiments that can be practiced or carried out in various ways, as well as combinations of the disclosed embodiments. It is also to be understood that the terms and terminology used herein are for the purpose of description and should not be regarded as limiting.

[0011] Unless otherwise indicated, and as will be apparent from the discussion below, in the discussion throughout this specification, use of terms such as "processing," "computing," "calculating," "determining," "enhancing," "deriving," and the like, will be understood to refer to operations and / or processes of a computer or computing system, or similar electronic computing device, that manipulate and / or transform data represented as physical quantities, e.g., electronic quantities, in the registers and / or memory of the computing system into other data similarly represented as physical quantities in the memory, registers of the computing system, or other device that stores, transmits, or displays such information.

[0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention improves the field of medical simulators by providing an efficient and economical method and mechanism for virtual reality training of airway management. Specifically, an airway management virtual reality training system is provided that uses a relatively simple and passive physical patient model to train a user (e.g., a trainee) in performing an airway management procedure. The patient model includes a modeled airway that is anatomically realistic, a head that can move side to side relative to the torso, and a jaw that can be opened and closed. The physical patient model includes electromagnetic sensors that provide the system data regarding the movement of parts of the model and the movement of tools in and out of the model.

[0013] A user or trainee can manipulate airway management tools such as a laryngoscope, manual resuscitator, and other tools to perform the airway management procedure, with pressure sensors on the tools providing input for the mechanical interaction of the airway management tools with parts of the physical patient model. The tools may be real tools, tool models, or multi-purpose elements simulated as specific tools in virtual reality (VR). Both the models and tools may have trackers that provide position and orientation data to the system. The system also has multiple VR sensors, such as cameras, to monitor the airway management procedure being performed by the trainee.

[0014] The system tracks and senses many parameters of the medical procedure being performed and uses them to provide the trainee with a continuous, detailed and consistent virtual reality representation, displaying the training situation in a realistic scene, utilizing and modifying the hierarchy of sensors to focus the representation on the key features of the training, and obtaining interactivity of the VR simulated patient model and evaluation of the trainee's performance.

[0015] Although the patient model in some embodiments is simple and passive compared to prior art training models, the VR system compensates for this simplicity with a realistic representation and display of the procedure being performed provided to the user (trainee). The system may be configured to have visual representations that suggest at least some of the tactile inputs provided by more complex prior art models. In particular, the data collected by the system is organized in a hierarchical and situation-related manner, thus providing a continuous and realistic VR representation of the procedure being performed, e.g., with respect to the position and orientation of the airway management tool relative to the modeled airway, with respect to the pressure applied by the airway management tool against the modeled airway, and with respect to the trainee's hand manipulating the tool.

[0016] To this end, the disclosed system dynamically identifies treatment situations performed by the trainee and dynamically manages a situation-related hierarchy among sensors to continuously and consistently maintain representations and displays for the identified treatment situations. For example, when a trainee operates or handles a manual resuscitator, the system can represent or display the trainee's hands hidden underneath based on data from the resuscitator's pressure sensor used to complement the visual data collected by the camera. In another example, a multi-purpose (e.g., general) tool may be used by the trainee to perform an assisting action and represented and displayed only in the VR scene as a corresponding specific real tool (e.g., scalpel, forceps, tubes such as ET (endotracheal) tubes, suction, stethoscope, capnometer, oximeter), further simplifying the system. The VR system can simulate scenes of multiple operations and multiple assistants who can monitor the trainee's interactions. For example, a camera can be used to sense the position and posture of the trainee and correlate them with the commands given by the trainee to a real or virtual helper. Furthermore, the trainee's hands can be represented even if they are outside the detection range of the camera by the VR system adjusting the VR representation according to the identified situation, e.g. showing the trainee's hand outstretched to receive a (simulated) tool. Embodiments of the disclosed system are illustrated in a non-limiting manner in the following figures:

[0017] One advantage of this system is the simplicity of the physical model, making it inexpensive to purchase and use, and robust for field training (e.g., military medicine). Another advantage is the reliable feedback it provides for a wide range of procedures and user actions, allowing trainees to perform a variety of airway management procedures in many environmental situations.

[0018] 1A and 1B are high level schematic block diagrams of an airway management training system 100 according to some embodiments of the present invention. Figures 2-4 are high level schematic diagrams of components of the airway management training system 100 according to some embodiments of the present invention. The airway management training system 100 includes a virtual reality (VR) system 150 that provides a trainee with a simulated scene 162 and instructions related to an airway management procedure that the trainee applies to a passive physical patient model 110 using a variety of real, modeled and / or simulated airway management tools 120.

[0019] As shown generally in FIG. 1A, the VR system 150 may be configured to dynamically identify treatment situations and respond according to a situation-related hierarchy of multiple sensors in the training system 100 to continuously provide a realistic representation of an airway management procedure applied by a trainee using the system 150.

[0020] The trainee may use a VR headset 102, possibly including an eye tracker (not shown) that provides the VR system 150 with data on the trainee's eye movements, which in turn provides the VR headset 102 with the VR representation 140. One or more trackers 104 may be attached to the VR headset 102 to track the trainee's head. The trainee may also use one or two gloves 103 (see FIGS. 1A and 1B), which may include additional trackers 104, such as on the trainee's hands and / or trainee's fingers, to track the position of the trainee's hands and fingers. Suitable gloves used may include Manus VR gloves or Noitom Hi5 VR gloves, or other suitable gloves. In certain embodiments, optical sensors 152, such as cameras, may be attached to the trainee's head (or headset 102), hands (or gloves 103), or other body parts to provide detailed images of the treatment procedure performed by the trainee. In particular embodiments, one or more gloves 103 may be configured to measure forces applied by the trainee while manipulating the physical model 110 (e.g., as shown diagrammatically in FIG. 2) and deliver the measurements to the VR system 150, which uses the measurements as additional sensor data. In particular embodiments, one or more gloves 103 may be configured to provide haptic feedback to the trainee and simulate additional structural features (such as the airway of the VR patient) by applying forces to the trainee's hand in addition to the forces experienced by manipulating the physical model 110, e.g., to enhance haptic simulation.

[0021] As shown diagrammatically in Fig. 1B, the VR system 150 may be configured to represent in a VR environment (140) a patient model 110 as a VR patient 141, a tool to be used 120 or a generic tool 120 as a corresponding VR tool 125, a trainee's hand 146 (and possibly other body parts of the trainee), a pressure 144 applied by the tool (the pressure VR representation may be visual as an indication, by means of using tactile cues or by other means), a surrounding scene 162, a virtual or real assistant 164, and medical equipment (not shown). Fig. 2 shows a physical training situation diagrammatically, Fig. 3 shows some details of the physical structure of the patient model 110 diagrammatically, and Fig. 4 shows some tools 120 diagrammatically.

[0022] 5A and 5B provide examples of a scene 162, a patient 141, tool representations 125B, 125A (of a manual resuscitator 120B and a laryngoscope 120A, respectively, the latter shown at the end of FIG. 5B and see further description below), and a VR representation 140 of a trainee's hand 146, according to some embodiments of the present invention. Each of FIGS. 5A and 5B includes an example of the actual VR representation 140 as well as line drawings showing the parts of the representations listed above. Continuous and consistent matching of the VR representation 140 to the trainee's movements on the patient model 110 allows for realistic training, effective learning, and reliable evaluation of the trainee's performance.

[0023] The airway management training system 100 includes a passive physical patient model 110 (e.g., a mannequin or part thereof) having a modeled airway 111, a head 112 connected to a torso 115 that can move from side to side, and a jaw 113 that can be opened and closed, as shown, for example, in Figures 2 and 3. The direction of movement of the head 112 is indicated generally by arrow 112A, and the direction of movement of the jaw 113 is indicated generally by arrow 113A. It should be noted that the simplicity of the patient model 110 (compared to mechanically complex prior art patient models that include multiple motors and associated actuators, more directions of movement, and more complex model features) is compensated for by the VR system 150, which replaces at least a portion of the haptic information in the prior art systems with visual information. Advantageously, the simplicity of the passive physical patient model 110 allows the patient model 110 to be robust and field deployable, for example, for training military and / or civilian medical personnel under realistic conditions.

[0024] The patient model 110 further includes one or more electromagnetic sensors 114 configured to measure the relative positions of the head 112, chin 113, and airway management tool 120 with respect to the modeled airway 111. For example, the electromagnetic sensors 114A, 114B (disposed within the head 112 as shown diagrammatically in FIG. 3) may be configured to measure the movement of the chin and head, respectively, to provide data 153 from the electromagnetic sensors, as shown diagrammatically in FIG. 2. FIG. 3 also shows a mechanical arrangement 112B (connecting the head 112 to the torso 115 and supporting rotational movement of the head 112) configured to enable side-to-side movement of the head 112 (movement 112A shown in FIG. 2). The modeled airway 111 is modeled to provide a realistic interaction with the applied tool 120, which, although not explicitly shown, corresponds to the anatomy of the airway. In certain embodiments, the patient model 110 may also include pressure sensors 117 placed at specific locations of importance during airway management to complement the pressure sensors 122 of the tool 120. The pressure sensors 122 of the tool 120, the optional flexion sensors 123, and the optical sensors 152 of the VR system 150 (and / or the tracker 104 of the headset 102) provide data 151 regarding the tool 120 that is used by the VR system 150 to identify treatment situations 156 performed by the trainee and to dynamically manage a situation-related hierarchy 158 of multiple sensors in the system 100.

[0025] The airway management tools 120 may include, for example, a laryngoscope 120A, a manual resuscitator 120B (e.g., an Ambu® resuscitator device or other resuscitator), and a multi-purpose tool 120C that may be represented or displayed in VR as any of a variety of tools (e.g., a scalpel, forceps, a tube such as an ET (endotracheal) tube, suction, stethoscope, capnometer, oximeter), as shown generally in FIG. The airway management tool 120 may include one or more pressure sensors 122 and (optionally) trackers 124 that provide feedback to the system 100 regarding the physical interaction between the tool 120 and the patient model 110, and feedback regarding the position and orientation 142 of the tool 120, which can be used, for example, to evaluate the trainee's performance and / or to enhance or modify the VR representation 140 of the procedure, generate a response from the simulated patient, etc.

[0026] In various embodiments, the laryngoscope 120A may be real or modeled and includes a pressure sensor 122 along the blade of the laryngoscope 120A that is used to provide feedback regarding the force exerted by the laryngoscope 120A against the modeled airway 111 (e.g., teeth, jaws, or internal portions of the airway) as the trainee manipulates or handles the laryngoscope 120A. The feedback may be translated into VR instructions such as simulated patient injury or reaction, and / or to assess the quality of the trainee's application of each airway management procedure.

[0027] In various embodiments, the manual resuscitator 120B may be real or modeled and includes pressure sensors 122 and / or flexion sensors 123 along at least a portion of its circumference and / or at its mouthpiece 122A. The pressure sensors 122, 122A may be used to provide feedback regarding the force applied by the manual resuscitator 120B against the modeled airway 111 (e.g., teeth or jaws 113) as the trainee manipulates the manual resuscitator 120B. The feedback may be translated into VR indications such as injury or reaction of the simulated patient and / or to evaluate the quality of the trainee's application of the respective airway management procedure. The degree of airtightness between the manual resuscitator 120B and the mouth of the patient model 110 may also be measured and shown in the VR representation 140 and / or indicated by modifying the reaction of the simulated patient in the VR representation 140 (e.g., a poor airtightness may result in poor or no chest movement when manipulating the manual resuscitator 120B).

[0028] Additionally, data from pressure sensors 122 may be used to enhance or modify the VR representation of the trainee, for example, data from pressure sensors 122 indicating that the trainee's hands are under manual resuscitator 120B and are not visible to optical sensors 152 of VR system 150 may be used to accurately or approximately represent and display hands 146 (despite at least one hand being at least partially obscured from view by manual resuscitator 120B) to enhance the continuity of, and realism of, VR representation 140. For example, VR system 150 may be configured to represent trainee's hands 146 in response to the detected pressure applied to and / or flexion of the circumference of manual resuscitator 120B.

[0029] In various embodiments, multi-purpose tool 120C, which may have a generic design, may be used to adjustably represent or display any of a variety of auxiliary tools, such as a scalpel, forceps, a tube such as an ET (endotracheal) tube, a suction device, a stethoscope, a capnometer, an oximeter, etc. For example, VR system 150 may be configured to virtually provide and display tool 120C as any of a scalpel, forceps, an ET tube, a suction device, a stethoscope, a capnometer, an oximeter, etc. according to a trainee's request (e.g., from a real or simulated assistant 164) and / or according to a dynamically identified treatment context 156.

[0030] 5A and 5B provide examples of a VR representation 140 of a scene 162, a patient 141, a tool 125, and a trainee's hand 146 according to some embodiments of the present invention. A virtual reality (VR) system 150 may be configured to provide a trainee (e.g., wearing a VR headset 102 having a tracker 104 attached thereto, as shown diagrammatically in FIG. 2 ) with a scene 162 including at least one patient 141 corresponding to a physical patient model 110, and a representation 140 of a medical procedure performed by the trainee on a passive physical patient model 110 using an airway management tool 120. The VR headset 102 may include a head-mounted device that provides the VR representation 140 to the trainee, and may include, for example, a display and a processor associated with a computing device 154 disclosed below, and in wired or wireless communication with the VR system 150. The VR headset 102 may further include a stereoscopic head-mounted display, provide sound, head movement and / or eye tracking sensors, and (possibly) correlating and associated controllers. The VR system 150 and VR headset 102 allow a trainee to be trained in airway management procedures within the virtual and controlled environment of the VR representation 140.

[0031] The VR representation 140 may include a tool representation 125 of the tool 120, an indication of at least one position and orientation 142 of the airway management tool 120 relative to the modeled airway 111 (including the tool representation 125) (see examples in FIGS. 5A and 5B), a representation of a pressure 144 applied by the airway management tool 120 relative to the modeled airway 111 (shown diagrammatically in FIGS. 1A and 1B), and a representation of a trainee's hand 146 manipulating one of the airway management tools 120. The position and orientation 142 of the tool 120 may be represented visually, for example, as measured by an optical sensor 152 and / or a tracker 124, as shown in the non-limiting examples provided by FIGS. 5A and 5B. The pressure 144 may be represented, for example, by an indicator and / or a resistance to movement of the tool.

[0032] The VR system 150 may include optical sensors 152 configured to track at least one position of the passive patient model 110 (e.g., using the tracker 116 shown in FIG. 2), track the airway management tool 120 (e.g., to derive its position and orientation), and track the trainee's hands. The VR system 150 may be further configured to dynamically identify treatment situations 156 performed by the trainee, and dynamically manage a situation association hierarchy 158 between the sensors 152, 114, 122 that continuously and consistently maintains the representation 140 with respect to the identified treatment situations. The VR system 150 is configured to receive data from the sensors 114, 122, 123 and the trackers 104, 116, 124 wired and / or wirelessly.

[0033] It should be noted that continuity of the VR representation includes continuity of the motion of the represented elements, without jumps or jerks that do not correspond to real-world motion. A context-relevant sensor hierarchy 158 may be used regarding the relative reliability of various sensors, to eliminate potential discontinuous representation of elements implied by sensors due to limited field of view, relative distance from the respective element, or less relevant sensing mode, with sensors higher in the hierarchy providing more reliable data.

[0034] It should be noted that the consistency of the VR representation includes the consistency of the position of the represented elements, without disappearance or significant changes that do not correspond to real-world movements. The situation-relevant sensor hierarchy 158 may be used to exclude potential appearance or disappearance of elements in the VR representation of elements implied by sensors due to limited field of view, relative distance from the respective element, or less relevant detection mode for sensors higher in the hierarchy, with respect to the relative reliability of the various sensors. For example, if a hand or part of it is hidden under the manual resuscitator 120B and disappears from the field of view of the optical sensor 152, data from the flexion sensor 123 may be used to provide a VR representation of this hand in order to make this hand appear in the VR representation 140 in an appropriate manner. Another example concerns commands given by the trainee, which may involve hand movements outside the detection range. In such cases, the representation of the hand may be supplemented by tracker data or estimated position.

[0035] Specific non-limiting examples of situation-related sensor hierarchies 158 are provided below in Table 2. In general, for each or some of the identified treatment situations 156, the VR system 150 has rules that determine which of the sensors and trackers in the system 100 are more reliable and which are less reliable with respect to the geometry of the treatment situation (e.g., elements that are obscured or extend beyond the sensing range) and / or with respect to sensing modality (e.g., pressure data is more reliable than optical data in certain situations). The situation-related sensor hierarchy 158 may be determined according to such rules for each or some of the identified treatment situations 156.

[0036] The VR system 150 may be configured to generate a patient representation 141 from the patient model 110 by any of a variety of VR modeling procedures, for example using a polygonal mesh and adding surface features (see, e.g., FIG. 3 for an example of a polygonal mesh and see FIGS. 5A and 5B for examples of added surface features). Note that the physical patient model 110 corresponds to an internal data model in the VR system 150 that is used to build a VR representation 140 of a patient 141 that is used as a real-world reference for the medical procedure applied by the trainee and displayed to the trainee via the VR headset 102. The VR system 150 (and / or airway management training system 100) is configured to further augment the VR representation 140 with a visual representation of the airway management tool 120 according to its position and orientation relative to the modeled airway 111, a representation and / or indication of the pressure 144 applied by the airway management tool 120 against the modeled airway 111, and a visual representation of the trainee's hands 146 manipulating the airway management tool.

[0037] Table 1 provides some non-limiting examples of sensors and data in airway management training system 100. Trackers typically have 6DoF degrees of freedom and may include any available tracker with corresponding performance.

[0038] [Table 1]

[0039] Table 2 provides some non-limiting examples of treatment situations, movements monitored, sensors used, and corresponding VR representations and feedback.

[0040] [Table 2]

[0041] Specific non-limiting examples are provided below relating to the situations set forth in Table 2 of the sensor hierarchy used to assess the identified treatment situation 156, the associated instructions and / or corresponding virtual patient simulation features, the expected trainee responses, and the actual trainee responses corresponding thereto.

[0042] For example, in preparation for treatment and / or in various stages of treatment, the VR representation 140 may include simulated patient movements, sounds, head movements, various breathing patterns shown in the head and chest area and relevant to the medical situation, chest movements, and substances such as liquids in or out of the patient's airways, etc. The VR representation 140 may include typical patient behaviors corresponding to various patient states, such as alertness, consciousness, partial or full lack of consciousness, suffocation, sedation, various breathing patterns, etc.

[0043] And, the VR representation 140 may be altered in response to the trainee's actions or inactions, such as specific instructions, diagnostic procedures such as procedures applied to the patient model, measurements and instructions taken, use of tools in diagnosis and treatment, etc. In particular, incorrect or incomplete application of tools to the patient model 110 may alter the VR representation 140 in a manner that reflects improper application, for example, incorrect use of the manual resuscitator 120B may cause the virtual patient to not respond as expected (e.g., not breathe as expected with respect to chest movement, head movement, and associated sounds) and therefore require correction by the trainee. It is noted that the virtual patient's responses may be represented in the VR representation 140 and / or in the associated medical data and indicators.

[0044] Examples of modifications of the sensor hierarchy 158 and the VR representation of the passive patient model 110 with respect to the dynamically identified treatment situation 156 include, for example, the following with respect to the sensors listed in Table 1. The system 100 may be configured to modify the sensor hierarchy 158 and / or the resolution of the VR representation 140 according to the particular factors associated with the identified treatment situation 156. For example, if the trainee performs a fine motoric action, the finger tracker 104 may receive a higher priority than other sensors, and the resolution of the VR representation 140 in the respective area may be increased. In another example, if the trainee issues commands and receives tools, the resolution of the VR representation 140 may be reduced and sensor priority is assigned to tracking the large-scale scene. An additional simulation of a virtual assistant may be added to the VR representation 140. If the trainee applies a manual procedure to the modeled airway 111, head 112, and jaw 113, the respective electromagnetic sensors 114 receive priority to affect the VR representation 140 of the applied procedure. When a trainee uses tool 120, hand and / or finger tracker 104 may receive priority when the procedure is performed externally and the hands and fingers are visible, pressure sensor 122 may receive priority for internal application of tool 120 (e.g., insertion of an ET tube or laryngoscope blade), and other sensors, such as flexion sensor 122 on manual resuscitator 120B, may receive priority when application is external but the hands cannot be easily tracked, for example when the hands are underneath the resuscitator. Alternatively or complementary, gesture recognition may be used to reinforce specific procedures applied by the trainee.

[0045] The airway management training system 100 and virtual reality system 150 may include a computing device 154 or a portion thereof, e.g., a processor (see, e.g., FIG. 7 below), configured to perform the disclosed procedures, to continuously adjust the system responses and the VR representation 140 to the trainee, and to manage the evaluation of the trainee's performance.

[0046] 6 is a high level flow chart illustrating an airway management training method 200 according to some embodiments of the present invention. The method steps may be performed on the airway management training system 100 described above, which is optionally configured to perform the method 200. The method 200 may be at least partially implemented by at least one computer processor. Certain embodiments include a computer program product including a computer readable storage medium having a computer readable program embodied thereon and configured to perform the relevant steps of the method 200 (see, for example, FIG. 7 below). The method 200 may include the following steps, in any order:

[0047] The airway management training method 200 may include training airway management within a virtual reality environment (step 205); using a passive physical patient model and at least one airway management tool represented within the virtual reality environment (step 210), where the physical patient model has a modeled airway, a head that can move left and right, and a jaw that can open and close, and the at least one airway management tool has at least one pressure sensor (step 210); measuring the relative positions of the head, chin and at least one airway management tool with respect to the modeled airway (step 220); and providing the trainee within the virtual reality environment (step 230) with a VR scene including at least one VR patient corresponding to the physical patient model (using a plurality of electromagnetic sensors associated with the physical patient model) and a VR representation of a medical procedure performed by the trainee on the patient model, including the at least one airway management tool and the trainee's hands. The VR representation may include a step of displaying at least one position and orientation of at least one airway management tool relative to the airway and the trainee's hands manipulating the at least one airway management tool (step 240), and a step of indicating visually and / or using tactile cues the pressure applied by the at least one airway management tool against the modeled airway (step 241).

[0048] The airway management training method 200 may further include tracking (step 222) at least one position of the physical patient model, the position and orientation of at least one airway management tool, and the trainee's hands.

[0049] The airway management training method 200 may further include dynamically identifying treatment situations performed by the trainee (step 224) and dynamically managing a situation-related hierarchy among the sensors that continuously and consistently maintains a VR representation of the identified treatment situations (step 226).

[0050] In certain embodiments, the at least one airway management tool includes a laryngoscope with at least one pressure sensor on its blade, and method 200 further includes providing VR feedback to the trainee regarding a measurement of the pressure applied by the laryngoscope blade against the modeled airway (step 242).

[0051] In certain embodiments, the at least one airway management tool includes a manual resuscitator with at least one pressure sensor at the mouthpiece and at least one pressure and / or flexion sensor around at least a portion of its circumference, and method 200 further includes a step (step 244) of providing VR feedback to the trainee regarding measurements of the pressure applied by the trainee to the manual resuscitator (and / or flexion of the manual resuscitator by the trainee) and an airtight seal between the manual resuscitator and the mouth of the passive patient model.

[0052] The airway management training method 200 may further include displaying or representing the trainee's hands in response to the detected pressure applied to and / or flexion of the circumference of the manual resuscitator (step 246).

[0053] The airway management training method 200 may further include adjustably displaying or representing the multi-purpose tool in the VR representation as at least one airway management tool, such as at least one of a scalpel, forceps, a tube, an ET (endotracheal) tube, a suction device, a stethoscope, a capnometer, and an oximeter (step 248).

[0054] 7 is a high level block diagram of an exemplary computing device 154 that may be used in embodiments of the present invention. Computing device 154 may include a controller or processor 173, which may be or include one or more central processing units / processors (CPUs), one or more graphics processing units (GPUs or general purpose GPUs-GPGPUs), chips, or any suitable computing or computational device, an operating system 171, memory 172, storage 175, input devices 176, and output devices 177. Airway management training system 100 and virtual reality system 150 may be or include a computer system, such as that shown in FIG.

[0055] Operating system 171 may be or include any code segment designed and / or configured to perform tasks including coordinating, scheduling, arbitrating, monitoring, controlling, or otherwise managing the operation of computing device 154, e.g., scheduling the execution of programs. Memory 172 may be or include, e.g., random access memory (RAM), read only memory (ROM), dynamic RAM (DRAM), synchronous DRAM (SD-RAM), double data rate (DDR) memory chips, flash memory, volatile memory, non-volatile memory, cache memory, buffers, short-term memory units, long-term memory units, or other suitable memory or storage units. Memory 172 may be or include multiple, possibly different, memory units. Memory 172 may store, e.g., instructions (e.g., code 174) for performing methods, and / or data such as user responses, interruptions, etc.

[0056] Executable code 174 may be any executable code, such as an application, a program, a process, a task, or a script. Executable code 174 may be executed by controller 173, possibly under the control of operating system 171. For example, executable code 174, when executed, may cause application execution, such as computer code generation or editing, or VR execution or estimation, according to embodiments of the present invention. Executable code 174 may be code generated in a manner described herein. One or more computing devices 154 or components of computing devices 154 may be used for various modules and functions described herein. Devices including components similar to or different from those included in computing device 154 may be used and may be connected to a network and used as a system. One or more processors 173 may be configured to perform embodiments of the present invention, for example, by executing software or code.

[0057] Storage device 175 may be or may include, for example, a hard disk drive, a floppy disk drive, a compact disk (CD) drive, a CD-recordable (CD-R) drive, a universal serial bus (USB) device, or other suitable removable and / or fixed storage unit. Data, such as instructions, code, VR model data, parameters, etc., may be stored in storage device 175 and loaded from storage device 175 into memory 172 where the data may be processed by controller 173. In some embodiments, some of the components shown in FIG. 7 may be omitted.

[0058] The input devices 176 may be or may include, for example, a mouse, a keyboard, a touch screen or pad, or any suitable input device. It is recognized that any suitable number of input devices may be operably connected to the computing device 154, as indicated by block 176. The output devices 177 may include one or more displays, speakers, and / or any other suitable output devices. It is recognized that any suitable number of output devices may be operably connected to the computing device 154, as indicated by block 177. Any applicable input / output (I / O) devices may be connected to the computing device 154, for example, a wired or wireless network interface card (NIC), a modem, a printer or facsimile machine, a universal serial bus (USB) device, or an external hard drive may be included in the input devices 176 and / or the output devices 177.

[0059] An embodiment of the invention may include one or more articles, such as a computer or processor non-transitory readable medium, such as a memory, a disk drive, or a USB flash memory, or a computer or processor non-transitory storage medium (e.g., memory 172 or storage device 175), that encode, contain, or store instructions, e.g., computer-executable instructions, that, when executed by a processor or controller, perform the methods disclosed herein.

[0060] Aspects of the present invention have been described above with reference to flowchart illustrations and / or partial illustrations of methods, apparatus (systems) and computer program products according to embodiments of the present invention. It will be understood that each part of the flowchart illustrations and / or partial illustrations, and combinations of parts of the flowchart illustrations and / or partial illustrations, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, a special purpose computer, or other programmable data processing apparatus to produce a machine, and the instructions executed via the processor of the computer or other programmable data processing apparatus generate means for implementing the functions / acts specified in the flowchart illustrations and / or partial illustrations or parts thereof.

[0061] These computer program instructions may also be stored on a computer readable medium that can instruct a computer, other programmable data processing apparatus, or other device to function in a particular manner to produce an article of manufacture, where the instructions stored on the computer readable medium include instructions that implement the functions / acts specified in the flowcharts and / or sub-diagrams or portions thereof.

[0062] Computer program instructions may also be loaded into a computer, other programmable apparatus, or other device and cause a series of operational steps to be executed on the computer, other programmable apparatus, or other device to generate a computer-implemented process, where the instructions executed on the computer or other programmable apparatus provide a process for implementing the functions / operations identified in the flowcharts and / or sub-diagrams or portions thereof.

[0063] The above flow charts and figures illustrate possible architectures, functions and operations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each part of the flow chart or sub-diagrams may represent a module, segment or part of code, including one or more executable instructions for implementing a particular logical function. It is to be noted that in some alternative implementations, the functions depicted in the parts may occur out of the order depicted in the figures. For example, two parts shown in succession may in fact be executed substantially simultaneously, and the parts may in some cases be executed in reverse order depending on the functionality involved. It is to be noted that each part of the sub-diagrams and / or flow charts, and combinations of parts of the sub-diagrams and / or flow charts, may be implemented by a dedicated hardware-based system that executes a particular function or operation, or a combination of dedicated hardware and computer instructions.

[0064] In the above description, the embodiments are examples or implementations of the invention. The various appearances of "one embodiment", "embodiment", "particular embodiment" or "some embodiments" do not necessarily all refer to the same embodiment. Various features of the invention may be described in the context of a single embodiment, and the features may be provided separately or in any suitable combination. Conversely, although the invention is described in the context of separate embodiments herein for clarity, the invention may also be implemented in a single embodiment. A particular embodiment of the invention may include features from different embodiments described above, and a particular embodiment may incorporate elements from other embodiments described above. The disclosure of elements of the invention in the context of a particular embodiment should not be construed as limiting their use to only that particular embodiment. Furthermore, the invention can be implemented or embodied in various ways, and it should be understood that the invention may be embodied in specific embodiments other than those outlined above.

[0065] The present invention is not limited to these figures or the corresponding description. For example, the flow does not have to proceed through or through each illustrated box or state or move in exactly the same order as illustrated and described. The meanings of technical and scientific terms used in this specification should be commonly understood by those skilled in the art to which the present invention belongs, unless otherwise defined. Although the present invention has been described with respect to a limited number of embodiments, they should not be interpreted as limitations on the scope of the present invention, but rather as some exemplifications of preferred embodiments. Other possible variations, modifications, and applications are also within the scope of the present invention. Therefore, the scope of the present invention should not be limited to what has been described, but should be limited by the appended claims and their legal equivalents.

Claims

1. A physical patient model having an airway, a head that can move left and right, and a jaw that can be opened and closed; a plurality of electromagnetic sensors configured to measure the relative position of the head, the chin, and at least one airway management tool with respect to the airway; a virtual reality (VR) system configured to provide the trainee with a plurality of VR representations including a VR representation of a scene, a VR representation of at least one patient corresponding to the physical patient model, a VR representation of the at least one airway management tool, and a VR representation of a trainee's hands handling the at least one airway management tool; Including, The airway management training system, wherein the plurality of VR representations are based at least in part on the relative positions measured by the plurality of electromagnetic sensors.

2. the virtual reality system includes optical sensors configured to track at least one position of the physical patient model, a position and orientation of the at least one airway management tool, and a hand of the trainee; 2. The airway management training system of claim 1, wherein the virtual reality system is configured to dynamically identify treatment situations performed by the trainee and dynamically manage a situation-related hierarchy among optical sensors that continuously and consistently maintains the VR representation of the scene with respect to the identified treatment situations.

3. The airway management training system of claim 1 , wherein the VR representation of the scene associates a visual representation of the at least one airway management tool with the airway.

4. The airway management training system of claim 1 , wherein the VR representation of the scene illustrates pressure applied by the at least one airway management tool to the airway.

5. 10. The airway management training system of claim 1, further comprising at least one airway management tool having at least one pressure sensor.

6. 6. The airway management training system of claim 5, wherein the at least one airway management tool includes a laryngoscope with the at least one pressure sensor on a laryngoscope blade.

7. 2. The airway management training system of claim 1, wherein the at least one airway management tool comprises a manual resuscitator with at least one pressure sensor and / or at least one bending sensor on at least a portion of a circumference of the manual resuscitator.

8. 6. The airway management training system of claim 5, wherein the at least one airway management tool includes a multi-purpose tool visually represented as at least one of a scalpel, forceps, a tube, an ET (endotracheal) tube, a suction device, a stethoscope, a capnometer, and an oximeter.

9. training airway management in a virtual reality (VR) environment; augmenting the VR environment with a physical patient model and at least one airway management tool, the physical patient model having an airway, a head that can move left and right, and a jaw that can be opened and closed; measuring the relative position of the head and chin and the relative position of the at least one airway management tool with respect to the airway using a plurality of electromagnetic sensors associated with a passive patient model; displaying to the trainee within the VR environment a plurality of VR representations, including a VR representation of a scene, a VR representation of a patient corresponding to the physical patient model, a VR representation of the at least one airway management tool, and a VR representation of a trainee's hands; Including, The method of airway management training, wherein the plurality of VR representations are based at least in part on the measured relative positions.

10. 10. The airway management training method of claim 9, further comprising tracking a position of at least one of the physical patient model, the at least one airway management tool, and the trainee's hands.

11. 10. The airway management training method of claim 9, further comprising the steps of dynamically identifying therapeutic situations performed by the trainee, and dynamically managing a context association hierarchy among sensors that continuously and consistently maintains a VR representation with respect to the identified therapeutic situations.

12. the at least one airway management tool includes a manual resuscitator with at least one pressure sensor and / or at least one flexion sensor on at least a portion of a circumference of the manual resuscitator; 10. The airway management training method of claim 9, further comprising providing VR feedback to the trainee regarding a measurement of pressure applied by the trainee against the manual resuscitator and an airtight seal between the manual resuscitator and the mouth of the passive patient model.

13. 10. The airway management training method of claim 9, further comprising visually representing a multi-purpose tool as the at least one airway management tool in an adjustable manner as at least one of a scalpel, forceps, a tube, an ET (endotracheal) tube, a suction device, a stethoscope, a capnometer, and an oximeter.

14. The airway management training method of claim 9, wherein the at least one airway management tool has at least one pressure sensor.

15. A physical patient model having an airway, a head that can move left and right, and a jaw that can be opened and closed; a plurality of sensors associated with the physical patient model, the plurality of sensors including at least one of: one or more electromagnetic sensors, one or more optical sensors, one or more pressure sensors, one or more flex sensors, and one or more tracking sensors or one or more trackers; a virtual reality (VR) system configured to provide a trainee with a VR representation of a scene, a patient, an airway management tool, and at least a portion of the trainee's hands handling the at least one airway management tool; Including, The VR system includes: using data from said plurality of sensors to continuously provide a realistic VR representation of an airway management procedure applied by said trainee; Dynamically identifying a treatment situation to be performed by the trainee and dynamically managing a situation-related hierarchy among the plurality of sensors to continuously and consistently maintain the VR representation with respect to the identified treatment situation. The airway management training system further comprises:

16. The airway management training system of claim 15, wherein the plurality of sensors are configured to measure a relative position of the head and chin and a relative position of the at least one airway management tool relative to the airway, and the VR representation is based at least in part on the measurements of the relative positions.

17. The airway management training system of claim 15, wherein the VR system is further configured to provide a VR representation of the spatial relationship between the at least one airway management tool and the airway, and / or a VR representation of the pressure applied to the airway by the at least one airway management tool.

18. The airway management training system of claim 15, wherein the at least one airway management tool includes a manual resuscitator having at least one pressure sensor and / or at least one flexion sensor on at least a portion of the circumference of the manual resuscitator.

19. The airway management training system of claim 15, wherein the at least one airway management tool includes a multi-purpose tool visually represented as at least one of a scalpel, forceps, a tube, an ET (endotracheal) tube, a suction machine, a stethoscope, a capnometer, and an oximeter.

20. The at least one airway management tool includes a laryngoscope having the at least one pressure sensor on a laryngoscope blade; 15. The airway management training method of claim 14, further comprising the step of providing the trainee with VR feedback relating to a measurement of pressure exerted by a blade of the laryngoscope against the airway.

Images