Cardiopulmonary bypass machine with augmented reality display

Abstract

This specification describes devices used during surgical procedures for the treatment of cardiac disease. For example, this specification describes techniques for monitoring the operation of a heart-lung machine and then presenting related readouts on a head-mounted display to provide an augmented reality visual display. For example, various sensors on and around the heart-lung machine, the patient, and / or the extracorporeal circuit can monitor the operation of a procedure using a heart-lung machine.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Application No. 62 / 928,773, filed October 31, 2019. The disclosure of the prior application is deemed to be part of (and is incorporated by reference into) the disclosure of this application.

[0002] The present disclosure relates to a heart-lung machine that provides sensor data to a user in an augmented reality visual display. [Background technology]

[0003] Hollow fiber oxygenators are utilized in extracorporeal circuits to meet the gas exchange needs of patients during medical procedures such as cardiopulmonary bypass surgery. Blood from the patient is drained by gravity or vacuum assisted venous drainage (VAVD) is used to obtain the flow rate necessary to maintain sufficient volume in the reservoir. A pump, such as a peristaltic or centrifugal pump coupled with a magnetic drive system, may be used in the main line of the circuit to pump blood from the reservoir, through the oxygenator, and finally back to the patient.

[0004] Augmented reality (AR) is an interactive experience of a real-world environment in which objects present in the real world are sometimes augmented with computer-generated sensory information across multiple sensory modalities, including vision, hearing, touch, somatosensation, and smell. The overlaid sensory information can be constructive (i.e., adding to the natural environment) or disruptive (i.e., occluding the natural environment). Summary of the Invention

[0005] In one aspect, the present disclosure is directed to a system for displaying heart-lung machine sensor data in augmented reality. The system includes a head-mounted augmented reality display with a display field in which a user can view a physical object within its field of view, the augmented reality display configured to render glyphs and render video within the display field such that when the user views the physical object within its field of view, the user is shown a display of the glyphs and video as a superimposition on the view of the physical object. The system includes a heart-lung machine configured to engage in a procedure to provide an extracorporeal blood flow circuit to a patient. The system includes at least one fine-grained sensor configured to sense a first parameter of operation of the heart-lung machine. The system includes at least one coarse-grained sensor configured to sense a second parameter of operation of the heart-lung machine. The system includes at least one video sensor for capturing video data of operation of the heart-lung machine. The system includes a controller comprising a hardware processor and computer memory. The system includes a data network communicatively coupling at least the fine-grained sensor and the coarse-grained sensor to the controller and further coupling at least the controller to the head-mounted augmented reality display. The controller is configured to receive a first parameter of operation of the heart-lung machine over the data network, receive a second parameter of operation of the heart-lung machine over the data network, receive video data of the operation of the heart-lung machine over the data network, send instructions to the augmented reality display to show an alphanumeric value as a glyph based on the first parameter of operation of the heart-lung machine, send instructions to the augmented reality display to show a graphic selected from a plurality of possible graphics as a glyph based on the second parameter of operation of the heart-lung machine, and send instructions to the augmented reality display to show a video based on the video data. Other systems, methods, devices, products, and software may be used.

[0006] Implementations may include some, all, or none of the following features: the coarse-grained sensor is configured to sense the second parameter by sensing a particular physical phenomenon, and the video sensor captures the video data by capturing the particular physical phenomenon; the coarse-grained sensor is a level sensor configured to sense a level of blood in a reservoir of the heart-lung machine, and the video sensor captures the video data indicative of the level of blood in the reservoir of the heart-lung machine; the fine-grained sensor is configured to sense the first parameter by sensing a particular physical phenomenon, and the video sensor captures the video data by capturing the particular physical phenomenon; the coarse-grained sensor is a line pressure sensor configured to sense pressure in a line of the heart-lung machine, and the video sensor captures the video data indicative of the level of blood in the reservoir of the heart-lung machine; the coarse-grained sensor is an air detection sensor configured to sense the presence of air in a line of the heart-lung machine, and the video sensor captures the video data indicative of the level of blood in the reservoir of the heart-lung machine. The coarse-grained sensor is an arterial pressure sensor configured to sense a mean arterial pressure of the heart-lung machine, and the video sensor captures video data indicative of a level of blood in a reservoir of the heart-lung machine. The coarse-grained sensor is a flow sensor configured to sense a flow rate of the heart-lung machine, and the video sensor captures video data indicative of a level of blood in a reservoir of the heart-lung machine. The coarse-grained sensor is a venous saturation sensor configured to sense a mean venous saturation of the heart-lung machine, and the video sensor captures video data indicative of a level of blood in a reservoir of the heart-lung machine. The plurality of possible graphics includes an altered graphic that corresponds to the selected graphic and differs from the selected graphic in at least color. The color is red, and the controller is configured to deliver the selected graphic when the second parameter is within a normal range and to deliver the altered graphic when the second parameter is outside of the normal range.The controller is configured to send the selected graphic at a first time and send the modified graphic at a second time, subsequent to the first time. The system further comprises a standalone monitor, the controller further configured to send instructions to the standalone monitor to display monitor data on the standalone monitor, at least a portion of the monitor data including one of the group consisting of an alphanumeric value, a selected graphic, and a video. The video sensor records the alphanumeric representation, and the controller is further configured to generate a corresponding computer readable value and send instructions to the augmented reality display to display the corresponding computer readable value. The controller is further configured to determine changes to the corresponding computer readable value and send instructions to the augmented reality display to display the changes to the corresponding computer readable value. The controller is further configured to determine changes to a second parameter and send instructions to the augmented reality display to display as a second glyph based on the changes to the second parameter. The augmented reality display comprises a light emitter configured to emit light into the transparent display field to render the glyph. The augmented reality display comprises a light emitter configured to emit light into a half-mirror viewing field to render the glyphs. The augmented reality display comprises one or more screens configured to display video to render the glyphs. The augmented reality display comprises a retina projector configured to project images into one or more eyes of a user to render the glyphs.

[0007] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention pertains.Methods and materials similar or equivalent to those described herein can be used to implement the present invention, but suitable methods and materials are described herein.All publications, patent applications, patents, and other references mentioned herein are incorporated by reference in their entirety.In case of conflict, the present specification, including definitions, will control.In addition, materials, methods, and examples are illustrative only and are not intended to be limiting.

[0008] The techniques described herein may be used to provide one or more benefits. For example, the technique of medical intervention may be improved. A heart-lung machine may be made that provides an augmented reality monitor to any other user, including a perfusionist or other clinician, that provides an overlay with important information regarding the operation of the heart-lung machine that is being used to provide the medical intervention to the patient. The augmented reality monitor may be head-mounted, which may allow the perfusionist to take their gaze away from the heart-lung machine while still maintaining awareness of important indicators and sensor readings that essentially require constant monitoring and / or rapid response. This may give the perfusionist greater physical freedom, faster response to critical events, and better visual display of the data. For example, because a particular reading is always in a particular location in the perfusionist's view, the perfusionist may develop the skill of attending to this part of their view and see the information faster than on a fixed screen that may be in different locations for the perfusionist based on where the perfusionist is standing, sitting, or moving around the operating room (away from the heart-lung machine).

[0009] Rapid response by the perfusionist can be crucial in any number of critical situations. As will be appreciated, the survival of the patient can often depend on what the perfusionist sees and how effectively the perfusionist can respond to the event. Thus, the techniques described herein can lead to better patient outcomes, including survival, by improving the techniques used by perfusionists or other care workers. In some cases, the characteristics of the heart-lung machine can change very rapidly, for example, the blood level in the reservoir can be rapidly drained, which can result in air being introduced into the patient's body. The use of this technology can alert the perfusionist before such a critical event.

[0010] In addition, the techniques described herein can provide different types of displays to the perfusionist based on the type of sensing available. With fine-grained sensors, computer-generated numbers and letters can be generated. However, when fine-grained sensors are not available, other suboptimal options can be used. For example, with coarse-grained sensors.

[0011] The details of one or more embodiments of the invention are set forth in the accompanying drawings and the description herein. Other features, objects, and advantages of the invention will be apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0012] [Figure 1] 1 is a schematic diagram of a patient undergoing open heart surgery while being supported using an extracorporeal circuit in accordance with certain embodiments provided herein. [Diagram 2] 1 is a schematic diagram of exemplary monitoring and display hardware useful for providing heart-lung machine sensor data in augmented reality. [Diagram 3] 1 is a schematic diagram of an exemplary augmented reality visual display. [Figure 4] 1 is a schematic diagram of an exemplary augmented reality visual display. [Diagram 5] 1 is a schematic diagram of an exemplary augmented reality visual display. [Figure 6] 1 is a block diagram of an exemplary data processing apparatus; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Like numbers refer to corresponding parts throughout.

[0014] This specification describes devices used during surgical procedures for the treatment of heart disease. For example, this specification describes techniques for monitoring the operation of a heart-lung machine and then showing related readouts on a head-mounted display to provide an augmented reality visual display. For example, various sensors on and around the heart-lung machine, the patient, and / or the extracorporeal circuit can monitor the operation of a procedure using the heart-lung machine. The controller can then generate the visual display to be shown (e.g., if an oxygen saturation of 79.9845% is sensed, the alphanumeric characters "8", "0", ".", "0" and "%" can be used). The head-mounted display can then render this visual display (e.g., glyphs and videos).

[0015] As shown in FIG. 1, various types of medical procedures may be performed on the patient 10 while the patient 10 is connected to a life-supporting heart / pulmonary bypass machine system 100. In this example, the patient 10 is undergoing open-heart surgery, during which the patient's 10 heart 12 and lungs are purposefully stopped functioning temporarily. However, the patient's 10 body continues to have a metabolic need to be supplied with oxygen-rich blood circulating during the medical procedure, and the heart / pulmonary bypass machine system 100 performs such a function. That is, as will be further described below, the heart / pulmonary bypass machine system 100 is connected to the patient 10 and performs the functions of the patient's 10 heart 12 and lungs so as to keep the patient 10 alive and healthy during the open-heart surgery. The heart / pulmonary bypass machine system 100 may be used for many different types of medical procedures. For example, medical procedures in which the cardiac / pulmonary bypass machine system 100 may be used include, but are not limited to, coronary artery bypass grafting, heart valve repair, heart valve replacement, heart transplants, lung transplants, ablation procedures, septal defect repair, congenital heart disease repair, aneurysm repair, pulmonary endarterectomy, pulmonary thrombectomy, and the like.

[0016] The heart / pulmonary bypass machine system 100 is typically set up and operated by a specially trained clinician called a perfusionist. The perfusionist forms part of a broader cardiovascular surgery team that includes a cardiac surgeon, anesthesiologist, and nurses. During a medical procedure using the heart / pulmonary bypass machine system 100, the perfusionist is entrusted with many responsibilities, most importantly ensuring that the patient 10 remains alive and healthy by operating the heart / pulmonary bypass machine system 100 to maintain blood flow to the patient's tissues and regulating the levels of oxygen and carbon dioxide in the patient's 10 blood. Other responsibilities of the perfusionist include, but are not limited to, administering blood products, administering anesthetics or drugs, measuring selected laboratory values ​​(such as blood counts), monitoring circulation, monitoring blood gases, monitoring anticoagulation, inducing hypothermia, and hemodilution. The perfusionist's responsibilities are diverse, dynamic, and crucial to achieving a successful outcome of the procedure performed on the patient 10 using the heart / pulmonary bypass machine system 100.

[0017] In the illustrated example, cardiac / pulmonary bypass machine system 100 includes components and subsystems, such as a heart-lung machine 110, an extracorporeal circuit 120, one or more temperature control systems 130, a blood monitoring system 140, a perfusion data management system 150, and a regional oximetry system 160. Some types of procedures using cardiac / pulmonary bypass machine system 100 may not require all of the components and subsystems shown. Some types of procedures using cardiac / pulmonary bypass machine system 100 may require additional components and / or subsystems that are not shown.

[0018] The extracorporeal circuit 120 is connected to the patient 10 and the heart-lung machine 110. Other systems, such as a temperature control system 130, a blood monitoring system 140, and a perfusion data management system 150, may also be arranged to interface with the extracorporeal circuit 120. The extracorporeal circuit 120 is connected to the patient 10 at the patient's heart 12. Oxygen-depleted blood (venous blood) from the patient 10 is extracted from the patient 10 at the patient's heart 12 using a venous catheter 121. As explained further below, the blood is circulated through the extracorporeal circuit 120 to receive oxygen and remove carbon dioxide. The oxygen-rich blood is then returned through the extracorporeal circuit 120 to the patient's heart 12 via an aortic cannula 129.

[0019] The extracorporeal circuit 120 may include at least a venous tubing 122 coupled to a venous catheter 121, a blood reservoir 123, a centrifugal pump 124, an oxygenator 125, an arterial filter 126, one or more air bubble detectors 128, and an arterial tubing 127 coupled to an aortic cannula 129. The venous catheter 121 and the venous tubing 122 are in fluid communication with the venous side of the circulatory system of the patient 10. The venous tubing 122 is also in fluid communication with an inlet to the reservoir 123. An outlet from the reservoir 123 is connected by tubing to an inlet of the pump 124. An outlet of the pump 124 is connected by tubing to an inlet of the oxygenator 125. An outlet of the oxygenator 125 is connected by tubing to an inlet of the arterial filter 126. An outlet of the arterial filter 126 is connected to the arterial tubing 127. One or more pressure transducers may be placed along the arterial tubing 127 to detect the heart / lung machine (HLM) system line pressure of the blood in the arterial tubing 127, which is measured by the heart / lung machine 110 and monitored by a perfusionist. The arterial tubing 127 is connected to an arterial cannula 129, which is in physical contact with the heart 12 and in fluid communication with the arterial side of the circulatory system of the patient 10.

[0020] Briefly, the extracorporeal circuit 120 operates by drawing venous oxygen-depleted blood from the patient 10 via a venous catheter 121 and drawing the venous blood into a reservoir 123 via a venous tube 122. In some cases, gravity is used to force or drain the blood from the patient 10 into the reservoir 123. In some cases, a vacuum is used to help the blood flow from the patient 10 into the reservoir 123. The goal is that at least some blood is maintained in the reservoir 123 at all times during a surgical procedure. Otherwise, if the reservoir 123 empties, air may be pumped into the extracorporeal circuit 120 and possibly into the vasculature of the patient 10. Such an outcome could be potentially fatal to the patient 10. Thus, the perfusionist is tasked with visually monitoring the level of blood in the reservoir 123. Additionally, a level detector may be included with the reservoir 123 to generate an alarm in response to detection of a low level condition in the reservoir 123. Additionally, one or more air bubble detectors 128 may be positioned at various sites along the extracorporeal circuit 120. Blood from the reservoir 123 is drawn therefrom by a pump 124. The illustrated embodiment includes a disposable centrifugal pump as the pump 124, but in some cases, a peristaltic pump from the heart-lung machine 110 is used instead. The pressure generated by the pump 124 propels the blood through an oxygenator 125. A perfusionist adjusts the pump 124 to operate as needed while avoiding operational problems such as negative cavitation, which may generate micro-air particles in the blood of the extracorporeal circuit 120. In the oxygenator 125, the venous blood is enriched with oxygen and carbon dioxide is removed from the blood. The oxygen-rich arterial blood now exits the oxygenator 125, travels through an arterial filter 126 to remove emboli, and is pumped through an arterial tubing 127 into the patient's heart 12 via an aortic cannula 129.The extracorporeal circuit 120 may also include tubing and other components to facilitate functions such as, but not limited to, draining blood that accumulates in the heart of the patient 10, providing surgical suction to maintain visibility of the surgical field, delivering cardioplegia fluid to the heart 12 of the patient 10 during the procedure, measuring blood parameters, removing air from the blood, hemoconcentrating, adding medication, taking blood samples, heating and cooling the blood, and the like.

[0021] During a surgical procedure using cardiac / pulmonary bypass machine system 100, various vital signs of patient 10 are measured and / or monitored. For example, the patient's mean arterial pressure ("MAP") may be measured. The patient's MAP is a parameter monitored by a perfusionist operating cardiac / pulmonary bypass machine system 100 to ensure that the cardiac / pulmonary bypass machine system 100 is functioning as required during a surgical procedure. In some cases, the MAP reading is displayed on an anesthesia system screen and / or displayed on an operating room screen. If the patient's MAP is outside of a desired range, the perfusionist may make adjustments to cardiac / pulmonary bypass machine system 100 to improve the patient's MAP.

[0022] The heart / lung bypass machine system 100 also includes a heart-lung machine 110. The heart-lung machine 110 is a complex system including multiple pumps, monitors, controls, user interfaces, alarms, safety devices, etc., all of which are monitored and operated / adjusted by a perfusionist during a surgical procedure. For example, the illustrated heart-lung machine 110 includes an arterial pump 111 (which can be a drive system for the illustrated disposable centrifugal pump 124 or a peristaltic pump), an aspiration pump 112, an exhaust / drainage pump 113, a cardioplegia fluid pump 114, and a cardioplegia fluid delivery pump 115. The heart-lung machine 110 may also include or be in conjunction with devices such as tube occluders, gas blenders, etc. Parameters of the heart-lung machine 110, such as the rotational speed and other parameters of each of the pumps, are set and adjusted by the perfusionist. For example, the speed of the arterial pump 111 is adjusted to maintain a desired level of blood in the reservoir 123 and provide the necessary level of blood circulation within the patient 10.

[0023] The cardiac / pulmonary bypass machine system 100 also includes one or more temperature control systems 130. In a first embodiment, the temperature control system 130 is used to heat and cool the patient's blood in the oxygenator 125 via a heat exchanger. Additionally, the temperature control system 130 is used to heat and cool the cardioplegia fluid being pumped to the heart 12 of the patient 10. Typically, the temperature control system 130 is used in a cooling mode during the procedure (to reduce metabolic demand) and then used to warm the blood and / or cardioplegia fluid as the surgical procedure nears its completion. The perfusionist is tasked with monitoring and adjusting the temperature control system 130 as needed during the surgical procedure.

[0024] The cardiac / pulmonary bypass machine system 100 also includes a blood monitoring system 140, as shown. The blood monitoring system 140 is used to monitor the extracorporeal blood of the patient 10 during the surgical procedure. The monitored parameters may include, but are not limited to, pH, pCO2, pO2, K+, temperature, SO2, hematocrit, hemoglobin, base excess, bicarbonate, oxygen consumption, and oxygen delivery. The perfusionist is tasked with monitoring the blood monitoring system 140 during the surgical procedure. In some cases, the perfusionist is required to adjust other components or subsystems of the cardiac / pulmonary bypass machine system 100 depending on the readings from the blood monitoring system 140.

[0025] The cardiac / pulmonary bypass machine system 100 also includes a perfusion data management system 150 and a regional oximetry system 160, as shown. These systems may also be used by a perfusionist to monitor the condition of the patient 10 and / or the condition of the cardiac / pulmonary bypass machine system 100 during a surgical procedure.

[0026] From the above description, it can be observed and understood that the perfusionist bears a great deal of vital responsibility during a surgical procedure using the heart / pulmonary bypass machine system 100.

[0027] With reference to FIG. 2, hardware 200 is used to monitor the operation of a heart-lung machine, such as that described with reference to FIG. 1. Hardware 200 includes a head-mounted augmented reality display 202, which may be referred to as "smart glasses" or "smart eyeglasses," among other names. For example, display 202 may take the form of a pair of glasses, a canopy, an open area, or a face shield that a user (e.g., a perfusionist) wears on his or her head or face. Display 202 includes a viewing field through which a user can see physical objects within its field of view, and may be referred to as a "non-occlusion" or "non-occlusion heads-up display (HUD)," among other names. For example, display 202 may include a transparent portion of glass, plastic, or similar transparent material through which light emitted from physical objects enters the user's eyes. In another example, display 202 may be a solid opaque device that completely or partially blocks the user's view, and may be referred to as an "occlusion" or "occlusion HUD," among other names. In such cases, the field of view may include one or more screens (e.g., light-emitting diode or LED screens) along with one or more cameras that capture video data of the user's viewpoint. The video is then rendered on the screens, providing the user with a field of view that resembles, and sometimes is imperceptibly similar to, the clear view of the physical environment.

[0028] In yet another example, the display 202 can include a retina projector configured to project an image directly onto one or more eyes of the wearer. In some cases, the display 202 with a retina projector can include a transparent portion of glass, plastic, or similar transparent material through which light emitted from a physical object enters the user's eye. In some cases, the display 202 with a retina projector can include one or more cameras that capture video data of the user's viewpoint. The video is then rendered and projected onto one or more eyes of the user, providing the user with a display field that is similar, possibly imperceptibly similar, to the clear view of the physical environment. In some implementations, the display 202 can be configured to take into account the visual impairment of the user. For example, the retina projector can be configured to provide a projection for a user with a clouded cornea or cataracts in a manner that is clear to such a user.

[0029] In yet another example, the display 202 may include a half-mirrored portion of glass, plastic, or similar transparent material, through which light emitted from physical objects enters the user's eye, while light is emitted onto a half-mirrored display field to render glyphs, etc.

[0030] The display 202 is configured to render glyphs and render video within the display field. For example, a light emitter can emit light within a transparent display field to show a reflection of light to the user. In another example where a screen is used to show video from a user's viewpoint, the glyphs and video can be shown superimposed on the viewpoint video. In either case, the display 202 shows a visual display of the glyphs and video as a superimposition on the view of the physical object.

[0031] The display 202 may also include other features. For example, a microphone and earpiece may be included in the display 202 for connecting to an intercom, cell phone, or other telecommunications device. This may allow the operator to communicate with people in the same facility or further away via the microphone and earpiece.

[0032] The sensors 204-216 sense parameters of the operation of the heart-lung machine (see FIG. 1). As will be appreciated, a variety of different sensors may be used depending on the particular configuration of the heart-lung machine. For example, a pressure sensor may be used to sense pressure in the tubing of the extracorporeal circuit, a thermocouple may be used to sense operating pressure, reservoir levels may be monitored via ultrasonic sensors or strain gauges, blood parameters of the subject may be monitored using a multifunction blood parameter monitoring device, electrical impedance sensors, optical sensors, etc. Other possible sensors include battery charge sensors, proximity sensors, vibration sensors, gyroscope sensors, blood sensors capable of sensing DO2, VO2, Hgb, and hematocrit, etc.

[0033] Each of the sensors 204-216 is exposed to environmental phenomena in or around the heart-lung machine and generates an electrical signal based on the phenomena. These electrical signals may be digital signals, analog signals, or a mixture of the two. Appropriate circuitry may be used to smooth, normalize, and filter these signals, as necessary. Some sensors 204-210 are referred to herein as "fine-grain sensors" and some sensors 212-216 are referred to as "coarse-grain sensors," although this may or may not be reflected in the physical configuration of the sensors or in the electrical signals. As will be appreciated, subsampling, supersampling, interpolation, and averaging processes may be used to manipulate the operation of the sensors. Indeed, some particular sensors may be used as both fine-grain and coarse-grain, either at the same time or at different times.

[0034] Optional video sensor 218 captures video of the operation of the heart-lung machine. For example, video sensor 218 can include a light-responsive sensor panel (e.g., a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) panel). Video sensor 218 can also include an illumination source, a physical shutter, a flexible mount that allows for adjustable viewpoints, and the like.

[0035] The controller 220 includes a hardware processor and computer memory. For example, the computer memory may store a set of instructions, either in a read-only format or in a read-write format, for execution by the processor. These instructions may include software, firmware, drivers, data files, etc., used in the operation of the controller.

[0036] The terminal 222 may include any type of display, including non-head mounted. The terminal 222 may include a free-standing display, optionally with input devices such as a keyboard, mouse, trackball, or touchpad, may include a display worn on the arm, wrist, shoulder, or the like. The terminal 222 may provide a relatively stationary or mobile access point for input and output activities to the user. The terminal 222 or elements thereof may be mobile, such as by being attached to a swing arm, sliding drawer, user, or the like, including being attached to a heart-lung machine.

[0037] The network interface 224 can communicate with external network locations. For example, the network interface 224 can provide access to a local area network, the Internet, or other peripheral devices. Elements of the hardware 200 can use the network interface 224 to communicate with other elements of the hardware 200 or other remote destinations.

[0038] The data network 226 communicatively couples the elements of the hardware 200. As will be appreciated, the data network 226 may include one or more logical networks and one or more physical media for data exchange. For example, an Ethernet network may connect the sensors 204-216, the video sensor 218, and the controller 220, while a wireless (e.g., WiFi or BlueTooth) network connects the controller 220 and the display 202.

[0039] The controller 220 can receive data from the sensors 204-216 and the video sensor 218 via the data network 226. For example, the sensor 204 can be a pressure sensor that continuously reads the pressure in the lines as the heart-lung machine operates. The sensor 204 can report a first parameter (e.g., a four-digit real number of the sensed pressure, mm / Hg) to the controller 220. The controller 220 can then use this as a fine-grained parameter and send instructions to the display 202 to show an alphanumeric value (e.g., a string of four real digits and "mm / Hg") as a glyph based on the first parameter of the operation of the heart-lung machine.

[0040] The sensor 216 can be a fluid level sensor that returns "true" if the fluid, possibly blood, in the reservoir of the extracorporeal circuit is greater than a given volume. For example, the sensor 216 can be mounted in the reservoir or on its exterior, and if there is enough blood to cover the sensor 216, the sensor 216 sends a "true" value to the controller 220 as a second parameter, and if there is not enough blood to cover the sensor 216, the sensor 216 sends a "false" value. Alternatively, in some cases, the sensor 216 can be an ultrasonic level sensor that does not come into contact with blood. Upon receiving a "true" value, the controller 220 can send instructions to the display 202 to show a graphic of the reservoir with a green background, and the display 202 can indicate that there is enough blood in the reservoir. The controller can then receive a "false" value, and then send instructions to the display 202 to show a graphic of the reservoir with a red background, and the display 202 can indicate that there is not enough blood in the reservoir, or nearly so. In some cases, more than one sensor 216 may be used to sense the fluid level in the reservoir. For example, but not limited to, a first sensor 216 may be placed at a level that will send a warning message if the fluid level falls below the position of the first sensor 216. A second sensor 216 may be placed lower than the first sensor 216 at a level that will stop the arterial pump of the heart-lung machine to avoid air entering the extracorporeal circuit that may occur if the reservoir is depleted of fluid. As will be appreciated, these graphics may be stored by the processor 220 prior to operation of the heart-lung machine and may be selected to quickly convey semantic information to the user (e.g., yellow is a caution warning indicating a worsening condition, green is good, and red or blinking / flashing red is bad).

[0041] In some embodiments, optional video sensor 218 may be fixed to face the same reservoir as that being sensed by sensor 216. Video sensor 218 may send video data of the reservoir to controller 220. Controller 220 may send instructions to display 202 to show a video based on the video data received from video sensor 218.

[0042] In some implementations, the information collected by the sensor 216 and / or the video sensor 218 may be used to generate other values. For example, in addition to being sensed and used to visually display the blood level as a value, the controller 220 may also identify when the value of the level is changing (e.g., increasing or decreasing) and may identify the rate of such change. The increase or decrease may be indicated in the form of alphanumeric characters, glyphs (e.g., up and down arrows), colors (e.g., red / green or yellow / blue). The rate of change may be indicated separately from the indication of the change (e.g., with an exclamation point next to the change indication) or by modifying the change indication (e.g., by causing it to flash or change in size).

[0043] In some implementations, the video sensor 218 may be tasked with sensing one or more different phenomena. For example, in some implementations, the video sensor 218 may be tasked with recording the alphanumeric output of, for example, a meter or display. For example, an instrument may generate a human readable output that includes alphanumeric characters but does not have data network capabilities. In such a case, the video sensor 218 may record the alphanumeric output and the controller 220 may generate a corresponding computer readable value from the recording. In some embodiments, two or more of the video sensors 218 may be included as components of the hardware 200. For example, in some embodiments, a first video sensor 218 may be used to monitor a reservoir and one or more additional video sensors 218 may be used to monitor a meter and / or display. In some embodiments, the hardware system 200 does not include a video sensor 218 for monitoring a reservoir. Instead, only one or more level sensors (e.g., sensor 216) may be used to detect the level of fluid in the reservoir. In some embodiments, the video sensor 218 is not included at all (although the hardware 200 may include other components, e.g., sensors 204-216). It should be appreciated that the hardware system 200 is flexible and extensible such that it may be configured and adapted to the needs of a particular user (e.g., a perfusionist).

[0044] In some implementations, the information collected by the video sensor 218 may be used to generate other values. For example, in addition to being used to record the alphanumeric output, the controller 202 may also identify when the value of the alphanumeric output is changing (e.g., increasing or decreasing) and may identify the rate of such change. Increasing or decreasing values ​​may be indicated in the form of alphanumeric characters, glyphs (e.g., up and down arrows), colors (e.g., red / green or yellow / blue), etc. The rate of change of value may be indicated separately from the indication of the change (e.g., with an exclamation point next to the change indication) or by modifying the change indication (e.g., by causing it to flash or change size).

[0045] As can be appreciated herein, both the sensor 216 and the video sensor 218 may be used to sense the same or related phenomenon, in this case the blood level in a reservoir of an extracorporeal circuit coupled to a heart-lung machine. In this example, the sensor 216 may only provide an indication of whether there is more or less blood in the reservoir than a particular, possibly certain, level. However, the perfusionist may benefit from access to more information than just that single display. Thus, the visual display of the video of the reservoir in the display 202 may provide uninterrupted information regarding the level of blood with only simple physical and / or mental focus on a portion of the augmented reality visual display. However, if the perfusionist is focused on another task, it may also be difficult for them to pay full attention to the video. Thus, a highly visible glyph in the form of a graphic that appears to change from green to red would attract the perfusionist's attention to indicate that the perfusionist should pay attention to the reservoir video.

[0046] As can be appreciated, such a configuration may be beneficial for hardware 200 where fine-grained sensing is beneficial but technically problematic. In some heart-lung machines, accurate measurement of fluid levels in the reservoir becomes complicated or impossible due to material selection, interactions of currents as fluid is added to and removed from the reservoir at various rates, environmental interference, etc. However, hardware 200 may be utilized to take advantage of the fact that video of the reservoir provides more than enough information to quickly understand the state of the reservoir. Similarly, hardware 200 may be advantageously used by a perfusionist as an add-on to an existing heart-lung machine that is not provided with hardware 200. That is, the manufacturer of the heart-lung machine may provide only some of sensors 204-216, and the perfusionist may desire or require more data than is provided. In such cases, the perfusionist (or a technician instead) may modify the heart-lung machine with hardware 200 to provide greater functionality to an older or less featureful heart-lung machine.

[0047] In some cases, the hardware 200 may be developed as an integral part of the heart-lung machine. That is, the elements of the heart-lung machine may include, for example, a housing, a bracket, and a chamber, including the sensors 204-216. In some cases, the hardware 200 may be made available as an optional upgrade to the heart-lung machine by the vendor or manufacturer of the heart-lung machine. That is, a designer of the heart-lung machine may design the heart-lung machine such that some or all of the hardware 200 is available as an optional add-on component that may or may not be purchased with the heart-lung machine. In some cases, the hardware 200 may be made available as an add-on system after-sales service. That is, the hardware may be sold to an owner who already owns a heart-lung machine manufactured by a manufacturer different from the manufacturer of the hardware 200. Other distribution options are possible.

[0048] One beneficial feature of the system 200 is that various elements of the system 200 may be geographically separated and / or multiple copies of some components may be provided. For example, the sensors 204-216 may be located in one location (e.g., a hospital in one country) with one display 202 worn by a clinician at that location, while a second display 202 worn by another user may be located in another location (e.g., a training center in another country). In such a case, both users may be visually presented with the same view or different views depending on the context. One use case of such a configuration is the training use case. A trainer at one physical location can "virtually walk into" a procedure being performed by a clinician who is far away, geographically speaking. Using remote communication (e.g., telephone, video conferencing), the trainer may assist the clinician in an actual procedure or a simulated instructional procedure. For example, the trainer can help the clinician learn how to use a technology, how to perform a particular procedure, etc. This may be particularly valuable, for example, in treating patients with very rare or complex conditions.

[0049] Consider a patient with an extremely rare condition for which there is only one clinician in the world with deep expertise and experience. If the patient is being treated in a different location than the specialist, the local clinician can collaborate with the remote specialist. The local clinician can physically interact with the patient and the heart-lung machine, all while receiving the unique input and guidance that the specialist can provide given the most up-to-date information provided by the system 200.

[0050] Additionally, new clinicians may be trained in the use of the cardiopulmonary bypass machine by a trainer without requiring the trainer and new clinician to be in the same location. This may advantageously allow a trainer to remain in a central training hub and train new clinicians on the same day who are far from one another and who could not travel one to the other on the same day.

[0051] Similarly, an expert clinician can share the system 200 with a junior clinician so that the junior clinician can observe how the expert clinician performs a procedure. In addition to the advantages below, the system 200 can include many remote displays 202 (or terminals 222) so that many junior clinicians can be trained simultaneously. This can also include many junior clinicians who are geographically separated.

[0052] System 200 can also be used for diagnostic, troubleshooting, and technical support purposes. For example, a technician installing a new component in a heart-lung machine can be connected to a remote technical expert who can guide the technician through the installation process, troubleshoot installation errors, etc.

[0053] As will be appreciated, these and other use cases can be provided without the time, expense, and hassle of travel. Additionally, system 200 can share one user's perspective with other users, allowing communication to occur that would never occur face-to-face. For example, an expert clinician can find out in real time that a junior clinician is spending too much time focusing their attention on one reading while they should be paying more attention to another aspect of the perfusion system. This type of learning and observation can advantageously improve clinical outcomes, training, and service operations of system 200 and related medical technologies.

[0054] Referring to Figure 3, a schematic diagram 300 of an exemplary augmented reality visual display is shown. In this example, the visual display is configured for a visor or goggles that provides a generally rectangular area with the user's nose removed. Glyph areas 302 and 304, represented by the repeated letter "g", are areas of the visual display in which the display device is configured to visually display one or more glyphs. Video area 306 is an area of ​​the visual display in which the display device is configured to visually display one or more videos. It will be appreciated that the number, shape, size, and location of the areas may vary from visual display to visual display and may be constrained by the technical limitations of the display device used.

[0055] 4, a schematic diagram of an exemplary augmented reality visual display 400 is shown. In this example, the glyph area is populated with fields for possible glyphs related to reservoir level, line pressure, air detection, mean arterial pressure, flow rate, and venous saturation average.

[0056] 5, a schematic diagram of an exemplary augmented reality visual display 500 is shown. In this example, the fields for reservoir level and air detection are populated with graphics, while the remaining glyph areas are populated with alphanumeric values.

[0057] The augmented reality visual display may be configured to show the expected range of data for a range of humans or other patients. For example, the ranges of values ​​described herein may be used as follows:

[0058] MAP: 0-400mmHg or completely off in range settings; Flow rate: 0~10L / min or completely off in range setting; Venous Saturation: 0-100% or off, and Line Pressure: 0~500mmHg or off.

[0059] For example, as a smaller 60kg woman: MAP: Normal value may be around 60mmHg, with range settings of 55mmHg for low alarm and 80mmHg for high alarm; Flow rate: Normal is set at 4L / min, Low alarm at 3L / min, and High alarm at 5L / min. Venous saturation: normal is 80%, with a low setting of 65% and a high setting of 90%; Line pressure: 180mmHg, with a low setting of 90mmHg and a high setting of 250mmHg.

[0060] For example, a larger 100kg man: MAP: Normal value may be around 75mmHg, with range settings of 60mmHg for low alarm and 90mmHg for high alarm; Flow Rate: Normal is set at 6L / min, Low Alarm at 3.5L / min, and High Alarm at 7.5L / min. Venous saturation: 70%, with a low setting of 60% and a high setting of 80%; Line pressure: 240mmHg, with a low setting of 100mmHg and a high setting of 350mmHg.

[0061] Referring to FIG. 6, a block diagram of an exemplary data processing apparatus 600 is shown. The system 600 includes a processor 610, a memory 620, a storage device 630, and an input / output device 640. Each of the components 610, 620, 630, and 640 may be interconnected using, for example, a system bus 650. The processor 610 may process instructions for execution within the system 600. In one implementation, the processor 610 is a single-threaded processor. In another implementation, the processor 610 is a multi-threaded processor. The processor 610 may process instructions stored in the memory 620 or on the storage device 630.

[0062] The memory 620 stores information within the system 600. In one implementation, the memory 620 is a computer-readable medium. In one implementation, the memory 620 is a volatile memory unit. In another implementation, the memory 620 is a non-volatile memory unit.

[0063] The storage device 630 is capable of providing mass storage for the system 600. In one implementation, the storage device 630 is a computer-readable medium. In various different implementations, the storage device 630 may include, for example, a hard disk device, an optical disk device, or some other mass storage device.

[0064] The input / output devices 640 provide input / output operations to the system 600. In one implementation, the input / output devices 640 may include one or more network interface devices, such as Ethernet cards, serial communication devices, such as RS-232 ports, and / or wireless interface devices, such as 802.11 cards. In another implementation, the input / output devices may include driver devices configured to receive input data and send output data to other input / output devices, such as keyboards, printers, and display devices 660. However, other implementations, such as mobile computing devices, mobile communication devices, set-top boxes, television client devices, etc., may also be used.

[0065] Embodiments of the subject matter and operations described herein may be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or in a combination of one or more of them.Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by or to control the operation of a data processing apparatus.

[0066] A computer storage medium can be or be included in a computer readable storage device, a computer readable storage substrate, a random or serial access memory array or device, or a combination of one or more of these. Further, a computer storage medium is not a propagating signal, but a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagating signal. A computer storage medium can also be or be included in one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

[0067] The operations described herein may be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0068] The term "data processing apparatus" encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, programmable processors, computers, systems on chips, or a combination of these. The apparatus may include special purpose logic circuitry, e.g., a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, e.g., code constituting a processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of these. The apparatus and execution environment may implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0069] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple cooperating files (e.g., files that store one or more modules, subprograms, or portions of code). A computer program can be deployed to be executed on one computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0070] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform operations by operating on input data and generating output. The processes and logic flows may also be performed by, and an apparatus may be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0071] Processors suitable for executing computer programs include, by way of example, both general-purpose and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a processor for performing operations according to the instructions, and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, e.g., magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive data from or transmit data to, or both. However, a computer need not have such devices. Furthermore, a computer may be embedded in another device, e.g., a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few. Suitable devices for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0072] To provide for interaction with a user, embodiments of the subject matter described herein may be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, e.g., a mouse or trackball, by which the user can provide input to the computer. Other types of devices may also be used to provide for interaction with the user, e.g., feedback provided to the user may be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, speech, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on the user's user device in response to a request received from the web browser.

[0073] An embodiment of the subject matter described herein may be implemented in a computing system that includes a back-end component, e.g., as a data server, or a middleware component, such as an application server, or a front-end component, e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internetwork (e.g., the Internet), and a peer-to-peer network (e.g., an ad-hoc peer-to-peer network).

[0074] A computing system may include users and servers. The users and servers are generally remote from each other and typically interact through a communications network. The relationship of users and servers arises by virtue of computer programs running on the respective computers and having a user-server relationship to each other. In some embodiments, the server sends data (e.g., HTML pages) to a user device (e.g., to display the data to a user interacting with the user device and to receive user input from the user). Data generated at the user device (e.g., results of user interaction) may be received from the user device at the server.

[0075] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features specific to certain embodiments of a particular invention. Some features described in the context of another embodiment may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as acting in some combinations and may also be initially claimed as such, one or more features from the claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0076] Similarly, although operations are shown in the figures in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown, or in sequential order, or that all of the illustrated operations be performed, to achieve desired results. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged in multiple software products.

[0077] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to certain embodiments of a particular invention. Some features described in the context of another embodiment may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, features may be described in this specification as acting in some combinations and may also be initially claimed as such, but one or more features from the claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0078] Similarly, although operations are shown in the figures in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown, or in sequential order, or that all of the illustrated operations be performed, to achieve desired results. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described herein should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single product or packaged in multiple products.

[0079] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order depicted, or sequential order, to achieve desirable results. In some implementations, multitasking and parallel processing may be advantageous. The invention as originally claimed in the present application is set forth below. [1] A system for displaying heart-lung machine sensor data in augmented reality, comprising: a head-mounted augmented reality display having a display field within which a user can view a physical object, the augmented reality display configured to render the glyphs and render the video within the display field such that when the user views a physical object within the field of view, the glyphs and video are presented to the user as a superimposition on a view of the physical object; a heart-lung machine configured for use in a surgical procedure to provide an extracorporeal blood flow circuit to a patient; at least one fine grain sensor configured to sense a first parameter of operation of the heart-lung machine; at least one coarse grain sensor configured to sense a second parameter of the operation of the heart-lung machine; at least one video sensor for capturing video data of the operation of the heart-lung machine; a controller comprising a hardware processor and computer memory; a data network communicatively coupling at least the fine-grain sensor and the coarse-grain sensor to the controller, and further coupling at least the controller to the head-mounted augmented reality display; wherein the controller: receiving, via the data network, the first parameter of the operation of the heart-lung machine; receiving, via the data network, the second parameter of the operation of the heart-lung machine; receiving, via the data network, the video data of the operation of the heart-lung machine; sending instructions to the augmented reality display to display an alphanumeric value as a glyph based on the first parameter of the operation of the heart-lung machine; sending instructions to the augmented reality display to show as a glyph a graphic selected from a plurality of possible graphics based on the second parameter of the operation of the heart-lung machine; and sending instructions to the augmented reality display to show a video based on the video data. [2] the coarse-grained sensor is configured to sense the second parameter by sensing a particular physical phenomenon; The system of any one of claims 1 to 7, wherein the video sensor captures video data by capturing the particular physical phenomenon. [3] the coarse-grained sensor is a level sensor configured to sense a level of blood in a reservoir of the heart-lung machine; The system of any one of claims 2 to 6, wherein the video sensor captures video data indicative of the level of blood in the reservoir of the heart-lung machine. [4] The fine-grained sensor is configured to sense the first parameter by sensing a particular physical phenomenon; The system of any one of [1] to [3], wherein the video sensor captures video data by capturing the specific physical phenomenon. [5] the coarse-grained sensor is a line pressure sensor configured to sense pressure in a line of an artificial heart-lung machine; The system of any one of claims 1 to 4, wherein the video sensor captures video data indicative of the level of blood in a reservoir of the heart-lung machine. [6] the coarse-grain sensor is an air detection sensor configured to sense the presence of air in a line of a heart-lung machine; The system of any one of claims 1 to 4, wherein the video sensor captures video data indicative of the level of blood in a reservoir of the heart-lung machine. [7] the coarse-grained sensor is an arterial pressure sensor configured to sense mean arterial pressure in a heart-lung machine; The system of any one of claims 1 to 4, wherein the video sensor captures video data indicative of the level of blood in a reservoir of the heart-lung machine. [8] The coarse-grained sensor is a flow sensor configured to sense flow in a heart-lung machine; The system of any one of claims 1 to 4, wherein the video sensor captures video data indicative of the level of blood in a reservoir of the heart-lung machine. [9] the coarse-grained sensor is a venous saturation sensor configured to sense a mean venous saturation in a heart-lung machine; The system of any one of claims 1 to 4, wherein the video sensor captures video data indicative of the level of blood in a reservoir of the heart-lung machine.

[10] The system of any one of [1] to [9], wherein the plurality of possible graphics includes an altered graphic that corresponds to the selected graphic and differs from the selected graphic at least in color.

[11] The color is red; The system of

[10] , wherein the controller is configured to send the selected graphic when the second parameter is within a normal range and to send the modified graphic when the second parameter is outside of a normal range.

[12] The system of

[11] , wherein the controller is configured to send the selected graphic at a first time and send the modified graphic at a second time after the first time.

[13] The system further comprises a stand-alone monitor; The system of any one of [1] to

[12] , wherein the controller is further configured to send instructions to the stand-alone monitor to display monitor data on the stand-alone monitor, at least a portion of the monitor data including one of the group consisting of the alphanumeric value, the selected graphic, and the video.

[14] the video sensor records an alphanumeric display; wherein the controller: generating a corresponding computer readable value; and sending instructions to the augmented reality display to indicate the corresponding computer-readable value.

[15] The controller: determining a change to the corresponding computer readable value; and sending instructions to the augmented reality display to indicate the change to the corresponding computer-readable value.

[16] The controller: determining a change to the second parameter; and sending instructions to the augmented reality display to show as a second glyph based on the change in the second parameter.

[17] The system of any one of [1] to

[16] , wherein the augmented reality display comprises a light emitter configured to emit light within a transparent display field to render the glyphs.

[18] The system of any one of [1] to

[17] , wherein the augmented reality display comprises a light emitter configured to emit light within a half-mirror display field to render the glyphs.

[19] The system of any one of [1] to

[18] , wherein the augmented reality display comprises one or more screens configured to display video to render the glyphs.

[20] The system of any one of [1] to

[19] , wherein the augmented reality display comprises a retina projector configured to project images into one or more eyes of the user to render the glyphs.

[21] A controller device for showing heart-lung machine sensor data in augmented reality, the controller comprising a hardware processor and computer memory, the controller comprising: receiving, via a data network, a first parameter of operation of the heart-lung machine; receiving, via the data network, a second parameter of the operation of the heart-lung machine; receiving, via the data network, video data of the operation of the heart-lung machine; sending instructions to an augmented reality display to display an alphanumeric value as a glyph based on the first parameter of the operation of the heart-lung machine; sending instructions to the augmented reality display to show as a glyph a graphic selected from a plurality of possible graphics based on the second parameter of the operation of the heart-lung machine; and sending instructions to the augmented reality display to show a video based on the video data.

[22] The first parameter is generated from sensing a particular physical phenomenon; The controller device described in

[21] , wherein the video data is generated by capturing the specific physical phenomenon.

[23] The sensing of a particular physical phenomenon is sensing a level of blood in a reservoir of the heart-lung machine; The controller device of

[22] , wherein the capture of the particular physical phenomenon is an indication of the level of blood in the reservoir of the heart-lung machine.

[24] The fine-grained sensor is configured to sense the first parameter by sensing a particular physical phenomenon; The controller device of any one of

[21] to

[23] , wherein the video sensor captures video data by capturing the specific physical phenomenon.

[25] The controller device of

[24] , wherein the sensing of a particular physical phenomenon is sensing line pressure in a line of an artificial heart-lung machine, and the capturing of the particular physical phenomenon is an indication of the level of blood in the reservoir of the artificial heart-lung machine.

[26] The controller device of

[25] , wherein the sensing of the specific physical phenomenon is sensing the presence of air in a line of an artificial heart-lung machine, and the capturing of the specific physical phenomenon is an indication of the level of blood in the reservoir of the artificial heart-lung machine.

[27] the sensing of a particular physical phenomenon is sensing of mean arterial pressure in a heart-lung machine; The controller device of

[24] , wherein the capture of the particular physical phenomenon is an indication of the level of blood in the reservoir of the heart-lung machine.

[28] the sensing of a particular physical phenomenon is sensing flow of a heart-lung machine; The controller device of

[24] , wherein the capture of the particular physical phenomenon is an indication of the level of blood in the reservoir of the heart-lung machine.

[29] the sensing of a particular physical phenomenon is sensing of mean venous saturation of a heart-lung machine; The controller device of

[24] , wherein the capture of the particular physical phenomenon is an indication of the level of blood in the reservoir of the heart-lung machine.

[30] A controller device as described in any one of

[21] to

[29] , wherein the plurality of possible graphics includes an altered graphic that corresponds to the selected graphic and differs from the selected graphic at least in color.

[31] The color is red; The controller device of

[27] , wherein the controller is configured to send the selected graphic when the second parameter is within a normal range and to send the changed graphic when the second parameter is outside of the normal range.

[32] The controller device of

[28] , further configured to send the selected graphic at a first time and send the modified graphic at a second time after the first time.

[33] A controller device as described in any one of

[21] to

[32] , further configured to send instructions to the stand-alone monitor to display monitor data on the stand-alone monitor, at least a portion of the monitor data including one of the group consisting of the alphanumeric value, the selected graphic, and the video.

[34] The video sensor records an alphanumeric display; wherein the controller: generating a corresponding computer readable value; The controller device of any one of

[21] to

[33] , further configured to send instructions to the augmented reality display to indicate the corresponding computer-readable value.

[35] The controller: determining a change to the corresponding computer readable value; and sending instructions to the augmented reality display to indicate the change to the corresponding computer-readable value.

[36] The controller: determining a change to the second parameter; The controller device of any one of

[21] to

[35] , further configured to send an instruction to the augmented reality display to show as a second glyph based on the change in the second parameter.

[37] A system for displaying heart-lung machine sensor data in augmented reality, comprising: a head-mounted augmented reality display having a display field within which a user can see a physical object within its field of view, the augmented reality display configured to render the glyph within the display field such that when the user sees a physical object within its field of view, the glyph is presented to the user as a representation of the glyph as a superposition on a view of the physical object; a heart-lung machine configured for use in a surgical procedure to provide an extracorporeal blood flow circuit to a patient; at least one fine grain sensor configured to sense a first parameter of operation of the heart-lung machine; at least one coarse grain sensor configured to sense a second parameter of the operation of the heart-lung machine; a controller comprising a hardware processor and computer memory; a data network communicatively coupling at least the fine-grain sensor and the coarse-grain sensor to the controller, and further coupling at least the controller to the head-mounted augmented reality display; wherein the controller: receiving, via the data network, the first parameter of the operation of the heart-lung machine; receiving, via the data network, the second parameter of the operation of the heart-lung machine; sending instructions to the augmented reality display to display an alphanumeric value as a glyph based on the first parameter of the operation of the heart-lung machine; and sending instructions to the augmented reality display to show as a glyph a graphic selected from a plurality of possible graphics based on the second parameter of the operation of the heart-lung machine.

[38] A controller device for showing heart-lung machine sensor data in augmented reality, the controller comprising a hardware processor and computer memory, the controller comprising: receiving, via a data network, a first parameter of operation of the heart-lung machine; receiving, via the data network, a second parameter of the operation of the heart-lung machine; sending instructions to an augmented reality display to display an alphanumeric value as a glyph based on the first parameter of the operation of the heart-lung machine; and sending instructions to the augmented reality display to show as a glyph a graphic selected from a plurality of possible graphics based on the second parameter of the operation of the heart-lung machine.

Claims

1. 1. A system for displaying heart-lung machine sensor data in augmented reality, comprising: a head-mounted augmented reality display having a display field within which a user can view a physical object, the augmented reality display configured to render the glyphs and render the video within the display field such that when the user views a physical object within the field of view, the glyphs and video are presented to the user as a superimposition on a view of the physical object; at least one first sensor configured to sense a first parameter of operation of the heart-lung machine; a plurality of level sensors each configured to sense a level parameter of fluid entering the reservoir during operation of the heart-lung machine relative to a corresponding predefined level, each corresponding predefined level of the level sensors being different from the corresponding predefined levels of the other level sensors; at least one video sensor for capturing video data of the reservoir during the operation of the heart-lung machine; a controller comprising a hardware processor and computer memory; a data network communicatively coupling at least the first sensor and the plurality of level sensors to the controller, and further coupling at least the controller to the head mounted augmented reality display; wherein the controller: receiving, via the data network, the first parameter of the operation of the heart-lung machine; receiving the level parameter over the data network; and receiving, via the data network, the video data during the operation of the heart-lung machine; sending instructions to the augmented reality display to display an alphanumeric value as a glyph based on the first parameter of the operation of the heart-lung machine; sending instructions to the augmented reality display to show as a glyph a graphic selected from a plurality of possible graphics based on the level parameter; and sending instructions to the augmented reality display to show a video based on the video data.

2. The system of claim 1 , wherein the first sensor is a line pressure sensor configured to sense pressure in a line of a heart-lung machine.

3. The system of claim 1 , wherein the first sensor is an air detection sensor configured to sense the presence of air in a line of a heart-lung machine.

4. The system of claim 1 , wherein the first sensor is an arterial pressure sensor configured to sense mean arterial pressure of a heart-lung machine.

5. The system of claim 1 , wherein the first sensor is a flow sensor configured to sense a flow rate of a heart-lung machine.

6. The system of claim 1 , wherein the first sensor is a venous saturation sensor configured to sense a mean venous saturation of a heart-lung machine.

7. The system of claim 1 , wherein the plurality of possible graphics includes an altered graphic that corresponds to the selected graphic and differs from the selected graphic in at least color.

8. the color is red, 8. The system of claim 7, wherein the controller is configured to send the selected graphic when at least one of the level parameters is within a normal range and to send the altered graphic when the at least one of the level parameters is outside of a normal range.

9. The system of claim 8 , wherein the controller is configured to send the selected graphic at a first time and send the modified graphic at a second time after the first time.

10. The system further comprises a stand-alone monitor; 10. The system of claim 1, wherein the controller is further configured to send instructions to the stand-alone monitor to display monitor data on the stand-alone monitor, at least a portion of the monitor data including one of the group consisting of the alphanumeric value, the selected graphic, and the video.

11. said video sensor records an alphanumeric display; wherein the controller: generating a corresponding computer readable value; The system of claim 1 , further configured to: send instructions to the augmented reality display to show the corresponding computer readable value.

12. The controller: determining a change to the corresponding computer readable value; 12. The system of claim 11, further configured to: send instructions to the augmented reality display to indicate the change to the corresponding computer readable value.

13. The controller: determining a change to the level parameter; The system of claim 1 , further configured to: send instructions to the augmented reality display to show as a second glyph based on the change in the level parameter.

14. The system of claim 1 , wherein the augmented reality display comprises a light emitter configured to emit light within a transparent display field to render the glyphs.

15. 15. The system of claim 1, wherein the augmented reality display comprises a light emitter configured to emit light into a half-mirror display field to render the glyphs.

16. The system of claim 1 , wherein the augmented reality display comprises one or more screens configured to display video to render the glyphs.

17. 17. The system of claim 1, wherein the augmented reality display comprises a retina projector configured to project images into one or more eyes of the user to render the glyphs.

18. 1. A controller device for showing medical device sensor data in augmented reality, the controller comprising a hardware processor and computer memory, the controller comprising: receiving, over a data network, a first parameter of operation of the medical device; receiving, via the data network, a plurality of level parameters sensed by a corresponding plurality of level sensors each configured to sense at least one of the level parameters of fluid entering the reservoir during operation of the medical device, each level sensor being different from a corresponding predefined level of the other level sensors; receiving video data of the operation of the medical device over the data network; sending instructions to an augmented reality display to show an alphanumeric value as a glyph based on the first parameter of the operation of the medical device; sending instructions to the augmented reality display to show as a glyph a graphic selected from a plurality of possible graphics based on the level parameter; receiving, over a data network, video data of the operation of the medical device; and and sending instructions to the augmented reality display to show a video based on the video data.

19. The video sensor records the alphanumeric display, wherein the controller: generating a corresponding computer readable value; 20. The controller device of claim 18, further configured to: send instructions to the augmented reality display to indicate the corresponding computer readable value.

20. The controller: determining a change to the corresponding computer readable value; 20. A controller device according to claim 18 or 19, further configured to: send instructions to the augmented reality display to indicate the change to the corresponding computer readable value.

21. The controller: determining a change to the level parameter; The controller device of claim 18 , further configured to: send an instruction to the augmented reality display to show as a second glyph based on the change in the level parameter.

22. 1. A system for displaying heart-lung machine sensor data in augmented reality, comprising: a head-mounted augmented reality display having a display field within which a user can see a physical object within its field of view, the augmented reality display configured to render the glyph within the display field such that when the user sees a physical object within its field of view, the glyph is presented to the user as a representation of the glyph as a superposition on a view of the physical object; a heart-lung machine configured to engage in surgery to provide an extracorporeal blood flow circuit to a patient; at least one first sensor configured to sense a first parameter of operation of the heart-lung machine; a plurality of level sensors each configured to sense a level parameter of fluid entering the reservoir during operation of the heart-lung machine relative to a corresponding predefined level, each corresponding predefined level of the level sensors being different from the corresponding predefined levels of the other level sensors; means for capturing video data of the reservoir during operation of the heart-lung machine; a controller comprising a hardware processor and computer memory; a data network communicatively coupling at least the first sensor and the level sensor to the controller, and further coupling at least the controller to the head mounted augmented reality display; wherein the controller: receiving, via the data network, the first parameter of the operation of the heart-lung machine; receiving, via the data network, the level parameters of the operation of the heart-lung machine; receiving, via the data network, the video data during the operation of the heart-lung machine; sending instructions to the augmented reality display to display an alphanumeric value as a glyph based on the first parameter of the operation of the heart-lung machine; and sending instructions to the augmented reality display to show as a glyph a graphic selected from a plurality of possible graphics based on the level parameter.

Images