Medical device speech and gesture control methods, apparatuses, and system

Abstract

Methods, apparatuses, and systems for medical device speech and gesture control are disclosed herein. An example apparatus includes an audio interface that converts received audio into text, for example. The audio interface uses a current operating statue of the medical device to determine a command from the text and whether the command is enabled. When the command is enabled, the audio interface transmits a command message to a response manager, which converts the command to a medical device command signal. The response manager then transmits the medical device command signal to a control engine of the medical device to change how the medical device is operating based on the received audio. The same medical device command signal is used for similar touch and / or gesture inputs such that the control engine does not need to be modified based on which input modalities are implemented on the medical device.

Description

MEDICAL DEVICE SPEECH AND GESTURE CONTROL METHODS, APPARATUSES, AND SYSTEMBACKGROUND

[0001] Renal replacement therapy (“RRT”) is a therapy that replaces the normal bloodfiltering function of the kidneys. It is used when the kidneys are not working well, which is known as kidney failure and includes acute kidney injury (“AKI”) and chronic kidney disease (“CKD”). RRT involves removal of water from the body of a patient suffering from kidney failure, as well as exchange of solutes with the body. One example of RRT is an extracorporeal blood therapy, in which blood is circulated outside of a patient and interfaced with one or more medical fluids. Modalities of extracorporeal blood therapy include hemodialysis (“HD”), hemofiltration (“HF”) and hemodiafiltration (“HDF”). Another example of RRT is peritoneal dialysis (“PD”), in which a medical fluid is infused into a peritoneal cavity of a patient to interface with the blood of the patient through a peritoneal membrane.

[0002] RRT is performed by a dialysis system which is formed by arranging one or more disposable components on a dialysis machine. Medical fluids used in HD and PD are commonly known as dialysis fluids. In HF, the medical fluid is known as replacement fluid, since it is infused into the blood of a patient to replace fluid removed during therapy. In HDF, both dialysis fluid and replacement fluid are used.

[0003] Extracorporeal blood therapy by HD, HF, or HDF is performed differently for treatment of patients with AKI compared to patients with CKD, by use of a different type of dialysis machine. Generally, compared to CKD patients, AKI patients are treated continuously over a longer period of time and at lower fluid flow rates. Such continuous treatment is commonly known as CRRT (“Continuous Renal Replacement Therapy”). To ensure precise and consistent monitoring and control of fluid removal, known as ultrafiltration, AKI machines are typically provided with scales that are used for measuring the weight of fresh treatment fluid and the weight of spent treatment fluid during therapy. CKD machines instead use flow meters or volumetric pumping to control ultrafiltration.

[0004] PD machines, also known as cyclers, may include at least one scale to measure the weight of fresh treatment fluid infused into the peritoneal cavity and the weight of spent treatment fluid withdrawn from the peritoneal cavity. Alternatively, cyclers may use volumetric pumps to control ultrafdtration.

[0005] In any of the above modalities, an automated dialysis machine is located in a medical center or a patient’s home. To program or change a treatment, a patient or a clinician currently has to physically press a control interface, such as buttons or a touchscreen. However, many patients receiving dialysis treatments may have reduced dexterity such that pressing the control interface can be difficult or prone to errors. Additionally, some patients may lay down during a dialysis treatment and may not want to sit up or otherwise reposition themselves to have to interact with the dialysis machine.

[0006] In addition to patients, clinicians may prefer not having to physically touch the control interface of a dialysis machine. For instance, a clinician may have to sterilize their hands every time they contact a dialysis machine. In other instances, clinicians may be setting up line sets or attending to the patient between physically programming a dialysis machine.

[0007] A need accordingly exists for a dialysis system that provides hands-free control.SUMMARY

[0008] Example systems, methods, and apparatuses are disclosed herein for the audio and / or gesture control of medical devices. An example apparatus includes an audio interface that converts received audio into text, a unique message, or signal. The audio interface uses a current operating statue of the medical device to determine a command from the text, unique message, or signal and whether the command is enabled. When the command is enabled, the audio interface transmits a command message to a response manager, which converts the command to a medical device command signal. The response manager then transmits the medical device command signal to a control engine of the medical device to change how the medical device is operating based on the received audio. The same medical device command signal is used for similar touch and / or gesture inputs such that the control engine does not need to be modified based on which input modalities are implemented on the medical device.

[0009] In light of the disclosure herein and without limiting the disclosure in any way, in a first aspect of the present disclosure, which may be combined with any other aspect listed herein, a dialysis machine includes a memory device storing audio rules. The audio rules specify (i) which audio commands are enabled based on a current state of the dialysis machine, and (ii) which dialysis machine command signal is to be selected based on an audio command and the current state of the dialysis machine. The dialysis machine also includes a display screen including buttons for entering commands, a control interface configured to render graphical user interfaces for display by the display screen and output a touch command indicative of a specified dialysis machine operation to be performed, a microphone, and an audio interface configured to convert audio signals from the microphone into an audio command indicative of a specified dialysis machine operation to be performed. The dialysis machine further includes a control engine configured to perform dialysis machine operations based on received dialysis apparatus command signals and a processor communicatively coupled to the memory device, the display screen, the control interface, the control engine, and the audio interface. The processor includes a response manager configured to receive one of (a) the touch command from the control interface, or (b) the audio command from the audio interface, and communicate with the control engine to determine a current state of the dialysis machine. When the audio command is received and the (i) audio rules indicate the audio command is enabled, the response manager determines the dialysis machine command signal that corresponds to the audio command using the (ii) rules and the current state of the dialysis machine. The response manager also transmits the dialysis machine command signal to the control engine.

[0010] In a second aspect of the present disclosure, which may be combined with any other aspect listed herein, the current state of the dialysis machine includes at least one of programming setup, fluid line setup, disinfection, fluid line priming, recirculation, blood return, treatment administration, and a service mode.

[0011] In a third aspect of the present disclosure, which may be combined with any other aspect listed herein, the current state includes an indication of the graphical user interface displayed on the display screen.

[0012] In a fourth aspect of the present disclosure, which may be combined with any other aspect listed herein, the control interface includes at least one of buttons, knobs, switches, control surfaces, dials, or a touchscreen integrated with the display screen.

[0013] In a fifth aspect of the present disclosure, which may be combined with any other aspect listed herein, the dialysis machine command signals specified by the audio rules are indicative of at least one of setting or changing a dialysis machine treatment parameter, starting a dialysis treatment, pausing a dialysis treatment, stopping a dialysis treatment, muting or silencing an alarm or alert, starting a priming operation, stopping a priming operation, acknowledging a treatment setup step is complete, starting a disinfection operation, initiating a call to a clinician, changing a language of the audio commands, changing a trigger word, selecting a feature or option that is provided within a user interface shown on the display screen, navigating between different user interfaces, powering off the display screen, or begin recording audio.

[0014] In a sixth aspect of the present disclosure, which may be combined with any other aspect listed herein, the audio command includes at least one of a command word, a command phrase, or a command word and parameter values.

[0015] In a seventh aspect of the present disclosure, which may be combined with any other aspect listed herein, the audio interface is configured to convert the audio signals from the microphone into the audio command by converting the audio signals into text, a unique message, or signal, comparing the text, unique message, or signal to the audio commands provided in the audio rules, and when there is a match or favorable comparison between the text, unique message, or signal and one of the audio commands, transmitting the matched or favorably comparing audio command to the response manager.

[0016] In an eighth aspect of the present disclosure, which may be combined with any other aspect listed herein, the audio rules additionally include at least one trigger word, and the audio interface is further configured to compare the text, unique message, or signal to the at least one trigger word, and when there is a match between the text, unique message, or signal and the at least one trigger word, compare the text, unique message, or signal or additional text, unique messages, or signals to the audio commands provided in the audio rules for converting the audio signals from the microphone into the audio command.

[0017] In a ninth aspect of the present disclosure, which may be combined with any other aspect listed herein, the audio interface is further configured to start a timer when the at least one trigger word is detected within the text, and compare the text, unique message, or signal or the additional text, unique messages, or signals to the audio commands until the timer reaches a predefined time limit.

[0018] In a tenth aspect of the present disclosure, which may be combined with any other aspect listed herein, the response manager is further configured to, when the audio command is received determine which button or option of the control interface corresponds to the audio command, and illuminate or otherwise highlight the button or option of the control interface for a specified duration.

[0019] In an eleventh aspect of the present disclosure, which may be combined with any other aspect listed herein, the dialysis machine includes at least one of a hemodialysis machine, a hemodiafiltration machine, a peritoneal dialysis machine, or a continuous renal replacement therapy (“CRRT”) machine.

[0020] In a twelfth aspect of the present disclosure, which may be combined with any other aspect listed herein, when the response manager receives the touch command from the control interface, the response manager is configured to determine the dialysis machine command signal that corresponds to the touch command using the (ii) rules, and transmit the dialysis machine command signal to the control engine.

[0021] In a thirteenth aspect of the present disclosure, which may be combined with any other aspect listed herein, when the audio command is received and the (i) audio rules indicate the audio command is disabled, the response manager is configured to at least one of discard the audio command, or generate an alert indicative of the audio command being discarded.

[0022] In a fourteenth aspect of the present disclosure, which may be combined with any other aspect listed herein, at least one of the control interface and the audio interface are included within the processor.

[0023] In a fifteenth aspect of the present disclosure, which may be combined with any other aspect listed herein, the dialysis machine further includes a gesture sensor and a gesture interface communicatively coupled to the gesture sensor. The gesture interface is configured toconvert video or data indicative of arm / hand movements into a gesture command that is indicative of a specified dialysis machine operation to be performed.

[0024] In a sixteenth aspect of the present disclosure, which may be combined with any other aspect listed herein, the response manager is further configured to receive the gesture command from the gesture interface, when the gesture command is received and gesture rules indicate the gesture command is enabled, determine the dialysis machine command signal that corresponds to the gesture command using the gesture rules and the current state of the dialysis machine, and transmit the dialysis machine command signal to the control engine.

[0025] In a seventeenth aspect of the present disclosure, which may be combined with any other aspect listed herein, the dialysis machine further includes at least one pump, pinch valve, or solenoid configured to control a flow of fluid to perform at least some of the dialysis machine operations. The control engine is configured to control the at least one pump, pinch valve, or solenoid based on the dialysis apparatus command signal.

[0026] In an eighteenth aspect of the present disclosure, which may be combined with any other aspect listed herein, a method includes receiving, in an audio interface of a dialysis machine from a microphone of the dialysis machine, an audio input, converting, via the audio interface, the audio input into text, a unique message, or signal, and determining, via the audio interface, whether the received text, unique message, or signal includes a specified trigger word. When the received text, unique message, or signal includes the specified trigger word, the method includes determining, via the audio interface, if a command word is included within text, unique message, or signal that is received within a time period from when the trigger word was received. Additionally, when the command word is received within the time period, the method includes using, via the audio interface, a data structure of audio rules and a current operating state of the dialysis machine to determine if an audio command corresponding to the command word is enabled. The method further includes transmitting an audio command message indicative of the audio command to a response manager of the dialysis machine when the command word is enabled, determining, via the response manager using the same data structure or a different data structure, a dialysis machine command signal that corresponds to the audio command, and transmitting from the response manager to a control engine of the dialysis machine, the dialysis machine command signal.

[0027] In a nineteenth aspect of the present disclosure, which may be combined with any other aspect listed herein, the method further includes when a command word is not received during the time period, discarding, via the audio interface, additionally received text, unique messages, or signals until the trigger word is received again.

[0028] In a twentieth aspect of the present disclosure, which may be combined with any other aspect listed herein, the method further includes causing at least one button or graphical button of the dialysis machine to illuminate or blink when the when the command word is enabled.

[0029] In a twenty-first aspect of the present disclosure, which may be combined with any other aspect listed herein, the method further includes, when the audio command is not enabled, discarding the command word and / or causing a message to be displayed indicative of the audio command being disabled.

[0030] In a twenty-second aspect of the present disclosure, which may be combined with any other aspect listed herein, a dialysis machine includes a memory device including gesture rules, the gesture rules specifying (i) which gesture commands are enabled based on a current state of the dialysis machine, and (ii) which dialysis machine command signal is to be selected based on a gesture command and the current state of the dialysis machine. The dialysis machine also includes a display screen including buttons for entering commands, a control interface configured to render graphical user interfaces for display by the display screen and output a touch command indicative of a specified dialysis machine operation to be performed, a gesture sensor, and a gesture interface communicatively coupled to the gesture sensor. The gesture interface is configured to convert video or data indicative of arm / hand movements into a gesture command that is indicative of a specified dialysis machine operation to be performed. The dialysis machine further includes a control engine configured to perform dialysis machine operations based on received dialysis apparatus command signals and a processor communicatively coupled to the memory device, the display screen, the control interface, the control engine, and the gesture interface. The processor includes a response manager configured to receive one of (a) the touch command from the control interface, or (b) the gesture command from the gesture interface, and communicate with the control engine to determine a current state of the dialysis machine. When the gesture command is received and the (i) gesture rules indicate the gesture command is enabled, the response manager is configured to determine the dialysis machine command signal that corresponds to the gesturecommand using the (ii) rules and the current state of the dialysis machine. Additionally, the response manager transmits the dialysis machine command signal to the control engine.

[0031] In a twenty-third aspect of the present disclosure, which may be combined with any other aspect listed herein, the gesture sensor includes at least one of a camera, an infrared sensor, a capacitive sensor, or a depth sensor.

[0032] In a twenty-fourth aspect of the present disclosure, which may be combined with any other aspect listed herein, when the gesture command is received and the (i) gesture rules indicate the gesture command is disabled, the response manager is configured to at least one of discard the gesture command, or generate an alert indicative of the gesture command being discarded.

[0033] In a twenty-fifth aspect of the present disclosure, which may be combined with any other aspect listed herein, the dialysis machine further includes a microphone and an audio interface electrically coupled to the microphone. The audio interface is configured to convert audio signals from the microphone into an audio command indicative of a specified dialysis machine operation to be performed. The response manager is further configured to receive the audio command from the audio interface, when the audio command is received and audio rules indicate the audio command is enabled, determine the dialysis machine command signal that corresponds to the audio command using the audio rules and the current state of the dialysis machine, and transmit the dialysis machine command signal to the control engine.

[0034] In a twenty-sixth aspect of the present disclosure, any of the structure and functionality disclosed in connection with Figs. 1 to 9 may be combined with any of the other structure and functionality disclosed in connection with Figs. 1 to 9.

[0035] In light of the present disclosure and the above aspects, it is therefore an advantage of the present disclosure to provide a dialysis machine that accepts audio and / or gesture inputs.

[0036] It is another advantage of the present disclosure to use a response manager to convert touch inputs, audio inputs, and / or gesture inputs into a common signal structure for a lower level processor that controls operations of a dialysis machine.

[0037] Additional features and advantages are described in, and will be apparent from, the following Detailed Description and the Figures. The features and advantages described herein are not all-inclusive and, in particular, many additional features and advantages will be apparent toone of ordinary skill in the art in view of the figures and description. Also, any particular embodiment does not have to have all of the advantages listed herein and it is expressly contemplated to claim individual advantageous embodiments separately. Moreover, it should be noted that the language used in the specification has been selected principally for readability and instructional purposes, and not to limit the scope of the inventive subject matter.BRIEF DESCRIPTION OF THE FIGURES

[0038] Fig. 1 shows a known command flow for a dialysis machine that includes a touchscreen.

[0039] Fig. 2 shows a response manager of a dialysis machine 200 is configured with input modalities for touch, audio, and gestures, according to an example embodiment of the present disclosure.

[0040] Fig. 3 is a diagram of the dialysis machine of Fig. 2 located in a medical system that includes a server and a user device, according to an example embodiment of the present disclosure.

[0041] Fig. 4 is a diagram of one or more data structures for providing dialysis machine commands based on touch, audio, and gesture inputs, according to an example embodiment of the present disclosure.

[0042] Fig. 5 is a diagram of the one or more data structures of Fig. 4 with correspondences between inputs and commands, according to an example embodiment of the present disclosure.

[0043] Fig. 6 illustrates the dialysis machine of Figs. 2 and 3 with a processor configured to process touch, audio, and / or gesture inputs, according to an example embodiment of the present disclosure.

[0044] Fig. 7 is a diagram of an example process performed by an audio interface of the dialysis machine to determine an audio command from recorded audio, according to an example embodiment of the present disclosure.

[0045] Figs. 8A to 8C show example gestures that may be used to control operation of the dialysis machine, according to example embodiments of the present disclosure.

[0046] Fig. 9 is an alternative embodiment of how touch, audio, and / or gesture inputs may be processed within the dialysis machine of Fig. 3, according to an example embodiment of the present disclosure.DETAILED DESCRIPTION

[0047] Methods, systems, and apparatuses are disclosed for touch, audio, and / or gesture control of medical devices, and in particular, dialysis machines. The methods, systems, and apparatuses are configured to integrate touch, audio, and / or gesture control with a medical device such as a dialysis machine without changing known command structures for button and / or touchscreen control. The methods, systems, and apparatuses use a response manager that is configured to convert commands to corresponding dialysis machine command signals such that a lower level control system processes the inputs regardless of the input modality (e.g., a touchscreen, gesture control, or audio). The response manager may also be configured with one or more rules that specify how certain commands are converted and / or discarded based on, for example, a current operating state of the medical device. For example, some gesture or audio controls may be disabled during a dialysis treatment to prevent inadvertent audio or other sounds from affecting the treatment.

[0048] Fig. 1 shows a known command flow for a dialysis machine 100 that includes a touchscreen 102. The dialysis machine 100 also includes a graphical user interface (“GUI”) response manager 104 that converts touch input commands received by the touchscreen 102. The response manager 104 converts the commands into dialysis machine command signals, which are transmitted to a lower level system or control engine 106 for controlling the dialysis machine 100. The control engine 106 may include or be configured within a control processor that operates one or more pumps, valves, sensors, etc. of the dialysis machine 100 using one or more dialysis control algorithms in response to received dialysis machine command signals.

[0049] The touchscreen 102 may include a capacitive screen that transmits coordinates of a user’s touch. The response manager 104 tracks which graphical user interface is currently being displayed and coordinates of graphical buttons that are included within the user interface. The response manager 104 uses the coordinates of the user’s touch to determine which graphical button is selected and a corresponding command. The response manager 104 then transmits a dialysismachine command signal to the control engine 106 to change an operation and / or setting of the dialysis machine 100.

[0050] An issue with the known dialysis machine 100 is that it is not capable of receiving audio and / or gesture controls. Many dialysis machines are home-based machines that are setup for critically ill patients who may not have the necessary dexterity in their arthritic hands to use the touchscreen 102. Additionally, many patients prefer to lay down during treatment. Sitting up to use the touchscreen 102 is not desirable. For center-based dialysis machines, clinicians typically interact with the touchscreen 102. However, the interaction may slow the setup time as the clinicians have to connect fluid tubing for the dialysis machine in between selecting the appropriate settings and treatment parameters on the touchscreen 102.

[0051] One possible solution to add audio control is to add a separate response manager 104 and interface that are specifically designed for audio commands. Additionally, the control engine 106 has to be updated to receive and process audio commands in addition to touchscreen inputs. As one can appreciate, this is a significant undertaking to add a new input modality. Further, there is a possibility of inconsistent operation between the touch input modality and the audio input modality when there are discrepancies between how the control engine 106 processes touch commands and audio commands. The level of complexity increases when gesture control is added. Additionally, for medical devices that require clearance by the Food and Drug Association (“FDA”), all of the input modalities have to be tested and meet specified requirements. In the aggregate, the above-reasons may deter medical device manufacturers from providing multimodality input control.

[0052] Fig. 2 shows a response manager 202 of a dialysis machine 200 that is configured with input modalities for touch, audio, and gestures, according to an example embodiment of the present disclosure. Touch commands are generated from a control interface 204 based on inputs received via a touchscreen and / or physical buttons of the dialysis machine 200. Audio commands are generated by an audio interface 206 based on audio commands received in a microphone of the dialysis machine 200. Gesture commands are generated by a gesture interface 208 based on gestures detected by a camera or movement sensor of the dialysis machine 200.

[0053] While Fig. 2 shows the dialysis machine 200 with three modalities, in some embodiments the dialysis machine 200 may only include two command modalities. For example,the dialysis machine 200 may only include the control interface 204 and the audio interface 206. Alternatively, the dialysis machine 200 may only include the control interface 204 and the gesture interface 206.

[0054] The control interface 204 includes a touchscreen and / or one or more buttons. When a touchscreen is included with the dialysis machine 200, the touchscreen may be part of a display screen that includes a capacitive touchscreen that transmits messages indicative of screen coordinates of a user’s touch. The messages are transmitted to a microcontroller and / or a processor. The microcontroller and / or a processor identify which graphical user interface is displayed by the display screen of the control interface 204 and coordinates of graphical buttons shown within the graphical user interface. Each graphical button is associated with a touch command for operating the dialysis machine 200. The microcontroller and / or processor matches the received touch coordinates to coordinates of one graphical button. Based on the match, the microcontroller and / or processor transmits a corresponding touch command message 205 to the response manager 202. In some embodiments, the microcontroller and / or a processor may be included as part of the response manager 202.

[0055] Physical buttons of the control interface 204 are electrically connected to pins of the microcontroller and / or processor. When a physical button is pressed, an electrical signal is input to the pin of the microcontroller and / or processor. Alternatively when the physical button is pressed, a voltage to the pin is pulled to ground. The microcontroller and / or processor determines which button was pressed based on which pin experienced the change in voltage. A lookup table or other data structure accessible to the microcontroller and / or processor correlates the button to one or more dialysis machine operations. In some instances, the same button may correspond to different operations based on a current operating state of the dialysis machine 200 and / or which graphical user interface is displayed. The microcontroller and / or processor determines which touch command should be generated using the current operating state of the dialysis machine 200 and / or which graphical user interface is displayed.

[0056] The audio interface 206 is configured to convert received audio to text, a unique message, or signal using one or more speech-to-text algorithms. The audio interface 206 also determines an audio command based on the text, unique message, or signal. The determination of the audio command may be based on a current operating state of the dialysis machine 200 and / ormenu options displayed by a display screen of the dialysis machine 200. The audio interface 206 may include a data structure or list that associates text, unique messages, or signals to one or more audio commands based on operating state and / or displayed menu options of the dialysis machine 200. The audio interface 206 matches the text, unique message, or signal identified from audio to closest matching text, unique messages, or signals within the data structure or list to determine the corresponding audio command message 207 to transmit to the response manager 202. It should be appreciated that in some embodiments the audio is not necessarily converted to text, but may instead be processed as a signal that is compared to known words and phrases.

[0057] The gesture interface 208 receives movement data from a camera, infrared sensor, and / or capacitive sensor of the dialysis machine 200. The gesture interface 208 determines a gesture or movement from the movement data. For example, a thumbs up gesture may be determined from recognizing a user’s hand in the shape of a fist with a single projecting digit corresponding to a thumb. A wave gesture may be determined by detecting a user’ s hand moving in one direction. The gesture interface 208 may include a data structure or list that associates detected gestures with gesture commands for transmitting gesture command messages 209 to the response manager 202. Similar to touch and audio commands, the determination of the gesture command may be based on a current operating state of the dialysis machine 200 and / or menu options displayed by a display screen of the dialysis machine 200.

[0058] As shown in Fig. 2, the response manager 202 is configured to receive each of the touch command messages 205 from the control interface 204, the audio command messages 207 from the audio interface 206, and / or the gesture command messages 209 from the gesture interface 208. The response manager 202 converts each of the different command types (corresponding to the same input) into a common dialysis machine command signal 211 for transmission to a control engine 210. In this manner, the response manager 202 transmits the same dialysis machine command signal 211 regardless of the input modality. As such, the control engine 210 does not need to be modified as input modalities are added or removed. Instead, only the response manager 202 is updated to reflect the input modalities that are part of the dialysis machine 200.

[0059] In an example, a display screen of the dialysis machine 200 displays a graphical user interface popup window that prompts a user to press ‘ok’ when certain dialysis tubing is connected to a patient’s catheter. Pressing ‘ok’ provides a configuration that the tubing isconnected and a treatment can begin. A user may touch a touchscreen at a location of an ‘ok’ graphical button to provide an input or select an ‘Accept’ physical button on the dialysis machine 200. In response to the input, the control interface 204 generates a touch command that is indicative of selecting an acceptance command, which is transmitted in a touch command message to the response manager 202. Alternatively, the user may vocally say ‘ok’, which is recorded by the audio interface 206. The ‘ok’ audio is converted to text, a unique message, or signal by the audio interface 206, which then determines the ‘ok’ text, unique message, or signal corresponds to an acceptance command. The audio interface 206 then transmits an audio command message to the response manager 202. Yet alternatively, the user may provide use their hand to provide a thumbs up gesture, which is recorded by the gesture interface 208. The example gesture interface 208 determines the detected thumbs up gesture corresponds to an acceptance command. The gesture interface 208 then transmits a gesture command message to the response manager 202. In either of the three input modalities, a user provides an acceptance of a prompt to indicating dialysis tubing is connected. The response manager 202 may potentially receive three different messages for the respective modalities. The response manager 202 is configured to determine that the three different messages correspond to the same dialysis machine command signal. As such, the control engine 210 operates in the same manner regardless of the input modality used.Example Dialysis Machine Embodiment

[0060] Fig. 3 is a diagram of the dialysis machine 200 of Fig. 2 located in a medical system 300 that includes a server 302 and a user device 304, according to an example embodiment of the present disclosure. The dialysis machine 200 is a medical device that is configured to accept one or more parameters specifying a treatment or prescription (i.e., treatment programming information). The dialysis machine 200 performs one or more dialysis routines, such as priming patient tubing, disinfecting tubing, and providing one or more dialysis treatments.

[0061] In the illustrated example, the dialysis machine 200 is the PrisMax CRRT machine manufactured by the assignee of the subject application. It should be appreciated that in other embodiments, the dialysis machine 200 may include any other renal failure therapy machine. Further, the dialysis machine 200 may instead be a medical device that includes an infusion pump, a physiological sensor, etc. For example, the medical device may include, for example, an infusionpump (e.g., a syringe pump, a linear peristaltic pump, a large volume pump (“LVP”), an ambulatory pump, multi-channel pump), a nutritional compounding machine, an oxygen sensor, a respiratory monitor, a glucose meter, a blood pressure monitor, an electrocardiogram (“ECG”) monitor, a weight scale, and / or a heart rate monitor. It should be appreciated that multimodality inputs may be used virtually on any medical device.

[0062] The example dialysis machine 200 may include a display screen 306 for displaying instructions and receiving control inputs from a user. The display screen 306 may include buttons, a control panel, or a touchscreen. The display screen 306 may also be configured to enable a user to navigate to a certain window or graphical user interface. The display screen 306 may also display instructions for operating or controlling the dialysis machine 200 and / or a status of the dialysis machine 200.

[0063] The example dialysis machine 200 also includes the control engine 210. The example control engine 210 operates according to one or more instructions or software routines 307 for performing a treatment on a patient. The instructions or software routines 307 may be stored on a memory device 308 of the dialysis machine 200. The memory device 308 may also store one or more graphical user interfaces 310 that are displayed by the display screen. The

[0064] Inputs for operating the dialysis machine 200 may be received by the display screen 306 via the control interface 204. The control engine 210 receives the inputs as dialysis machine command signals. The control engine 210 uses the inputs in conjunction with the instructions or software routines 307 to control operation of the dialysis machine 200. The control engine 210 also monitors device components for issues, which are documented as diagnostic information. The control engine 210 creates medical device data in conjunction with operating one or more pumps or other components to administer a treatment.

[0065] The dialysis machine 200 also includes a processor 312, which may include a microcontroller, an application specific integrated circuit, a logic controller, etc. The processor 312 may include the control interface 204, the audio interface 206, and the gesture interface 208 of Fig. 2. The processor 312 is configured to determine commands from touch, audio, and / or gesture inputs. The processor 312 uses one or more data structures 314 in the memory device to determine an association between the command and the dialysis machine command signal that is transmitted to the control engine 210.

[0066] Fig. 4 shows a diagram of the one or more data structures 314, according to an example embodiment of the present disclosure. In the illustrated example, the data structure 314 is organized by dialysis machine command signals 211. In this example, the data structure 314 lists which commands are associated with which signals 211. For example, the data structure 314 identifies gesture commands 209a, audio commands 207a, and touch commands 205a that correspond to a first dialysis machine command signal 211a. Reception of either of the commands 205a, 207a, and 209a through the respective input modality causes the first dialysis machine command signal 21 la to be selected for transmission to the control engine 210. While only one command signal is shown for each modality, a modality can include multiple command signals.

[0067] The data structure 314 also includes rules 402 for the command signals. The rules 402 define conditions when the command signals may be disabled and / or enabled based on an operating state of the dialysis machine 200. For example, when the audio command 207a is received and the dialysis machine 200 is in a certain operating state (e.g., setup), the rule 402 specifies that the audio command 207a is enabled, thereby enabling the corresponding dialysis machine command signal 211 to be transmitted. However, when the dialysis machine 200 is operating in a different state, such as performing a dialysis treatment, the audio command 207a may be disabled. As such, the rule 402 ensures the dialysis machine command signal 211 is not transmitted. Some commands may be disabled during treatment to prevent inadvertent audio from changing a treatment. For example, the rules 402 may enable a user to audibly pause a dialysis treatment but not stop a treatment. Further, commands for silencing some alarms may be disabled during treatment, such as alarms related to potential line occlusions or low dialysis fluid supplies. In some instances, all commands may be disabled, such as during software updates or when the dialysis machine 200 is powering off.

[0068] In some embodiments, the rules 402 may specify when a microphone of the dialysis machine 200 is turned off. For example, the rules 402 may specify that the microphone is to be turned off after a treatment has started. Another rule 402 may specify that the microphone is turned on when an alert or alarm is generated. In these embodiments, turning off the microphone prevents the audio interface 206 from receiving audio, thereby disabling audio control for the dialysis machine. The rules 402 may specify similar operations for a sensor that detects gestures.

[0069] In addition to operating status, at least some of the commands in the data structure 314 may be selected based on the current graphical user interface 310 that is shown on the display screen 306. For example, a first graphical user interface 310 may include data fields for entering treatment parameters for the dialysis fluid. An audio command corresponding to a rate is selected for the dialysis fluid based on the inclusion of the dialysis fluid rate within the first graphical user interface 310. A second graphical user interface 310 may include data fields for entering treatment parameters for an extracorporeal circuit, which may include a blood flow rate. Here, an audio command corresponding to a rate is selected for the blood flow based on the inclusion of the blood flowrate within the second graphical user interface 310.

[0070] The command signals may be indicative of at least one of setting or changing a dialysis machine treatment parameter, starting a dialysis treatment, pausing a dialysis treatment, stopping a dialysis treatment, muting or silencing an alarm or alert, starting a priming operation, stopping a priming operation, acknowledging a treatment setup step is complete, starting a disinfection operation, initiating a call to a clinician, changing a language of the audio commands, changing a trigger word, selecting a feature or option that is provided within a user interface shown on the display screen, navigating between different user interfaces, powering off the display screen, or begin recording audio. The treatment parameters may include an ultrafiltration removal rate, a treatment time, a dialysis fluid flow rate, a dialysis fluid dextrose concentration, a blood flow rate, a heparin infusion rate, and / or a blood substitution infusion rate. The command includes at least one of a command word, a command phrase, and / or a command word and parameter values. The dialysis machine command signals 211 correspond to digital messages and / or analog signals provided in a specified format as inputs to the control engine 210 for causing the dialysis machine 200 to perform the commanded operation.

[0071] In some embodiments, the data structure 314 may also provide a correspondence between received signals, text, or identified gestures, and the corresponding commands. Fig. 5 is a diagram of the one or more data structures 314 with correspondences between inputs and commands, according to an example embodiment of the present disclosure. In the illustrated example, the audio command 207a is correlated to a library of text, shown as TEXT 1 to TEXT 3. Text, a unique message, or signal that is generated from audio by the audio interface 206 is matched to the corresponding audio command 207a in the data structure 314. In someembodiments, an operating state of the dialysis machine 200 and / or the graphical user interface 310 shown by the display screen 306 is used to select or fdter the potential audio commands. The received text, unique message, or signal is then compared to the library of text, unique messages, or signals for each command to determine a match.

[0072] For gestures, the data structure 314 may include lists of gesture types that correspond to commands. The gesture types are identified by the gesture interface 208 by analyzing video and / or movement data. Similar to audio text, the corresponding command may be filtered or otherwise selected based on a current operating state of the dialysis machine 200 and / or the graphic user interface 310 that is displayed by the display screen 306.

[0073] The processor 312 is also configured to render the graphical user interfaces 310 for display on the display screen 306 of the dialysis machine 200. The graphical user interfaces 310 may include links between each other to provide a menu structure. Further, the graphical user interfaces 310 may specify conditions when certain of the interfaces are to be displayed, such as which the graphical user interfaces 310 are displayed at startup. The processor 312 determines which the graphical user interface 310 is to be displayed based on received commands and / or a current operating status of the dialysis machine 200. The processor 312 then renders the determined the graphical user interface 310 for display on the display screen 306.

[0074] As shown in Fig. 3, the processor 312 is communicatively coupled to the control engine 210, which may comprise another processor of the dialysis machine 200. The processor 312 may be separated from the control engine 210 to prevent issues at the processor 312 from affecting the operation of the dialysis machine 200. Alternatively, the processor 308 may include or be integrally formed with the control engine 210.

[0075] The example processor 312 of the dialysis machine 200 is communicatively coupled to the server 302 via a network 320. The example network 320 may include a local area network, a wide area network (e.g., the Internet), a cellular network, or combinations thereof. The server 302 may include a processor, a workstation, or a distributed cloud computing system.

[0076] The example server 302 is configured to provide management and updates for the touch, speech, and / or gesture control provided by the processor 312. The server 302 may, in some instances, remotely enable one or more of the input modalities. The server 302 may also update the data structures 314 with associations between input commands 205, 207, and 209 and thedialysis machine command signals 211. The server 302 may also update rules included within the data structures 314 that specify which commands are enabled, disabled, or correspond to certain machine operating conditions.

[0077] In some instances, the server 302 may also receive feedback from the processor 312 regarding use of the different input modalities. The feedback may include events corresponding to instances where a user un-did an executed command, which may be indicative that the user expected one result and the dialysis machine 200 operated in a different manner. The server 302 may use the feedback to update the data structures 314 to more accurately interpret audio and / or gesture inputs. In some instances, the server 302 may use machine learning to correlate audio and / or gestures with corresponding commands 307 and 309 and / or dialysis machine command signals 311.

[0078] In some embodiments, at least some operations described in connection with the processor 312 may instead be performed by the server 302. For example, the audio interface 206 and / or the gesture interface 208 may be configured on the server 302. In this example, the processor 312 transmits text, a unique message, or a signal of recorded audio to the server 302. The audio interface 206 at the server 302 is configured to determine the audio command 307 from locally stored data structures 314. The server 302 also determines the dialysis machine command signal 211 from the determined command. The server 302 then transmits the corresponding dialysis machine command signal 211 to the control engine 210 via the processor 312. A similar message flow may occur for video and / or movement data corresponding to a gesture input at the dialysis machine.

[0079] In some embodiments, the system 300 of Fig. 3 includes the user device 304, which may include a smartphone, a tablet computer, a desktop computer, a workstation, a smartwatch, smarteyewear, etc. The user device 304 is communicatively coupled to the processor 312 of the dialysis machine 200 via the network 320. In other instances, the user device 304 is communicatively coupled to the dialysis machine 200 via a local connection, such as Bluetooth®, Zigbee®, and / or near field communication (“NFC”).

[0080] In clinical environments, the user device 304 enables a clinician to remotely change settings of the dialysis machine 200 using touch, audio, and / or gesture inputs. This may be beneficial when a patient is located in a containment room that isolates apatient’s disease. Insteadof donning protective equipment, the clinician can provide audio inputs into a microphone of the user device 304 from outside the patient’s room. The user device 304 transmits text, a unique message, or signal corresponding to the spoken inputs to the dialysis machine 200 for processing.

[0081] In home environments, the user device 304 may be a remote control for the dialysis machine 200. Instead of having to move close to the dialysis machine 200 to audibly provide an input, a user can speak into a microphone of the user device 304, which transmits text, a unique message, or signal of the audio input to the dialysis machine 200 for processing. Such a configuration may enable a user to provide inputs while laying down when they are not within speaking range of the dialysis machine 200 or the user is soft spoken.

[0082] The user device 304 includes a processor 340 and a memory device storing machine-readable instructions 342. Execution of the machine-readable instructions 342 by the processor 340 causes an application 344 to be executed. The application 344 is configured to enable a user to remotely enter inputs to the dialysis machine 200. When connected to the dialysis machine 200 after an authentication process, the application 344 may display the same graphical user interfaces 314 that are displayed on the display screen 306. A user may select graphical buttons via the application 344 as though they were interfacing directly with the control interface 204 and display screen 306 of the dialysis machine 200. The application 344 is configured to transmit an identification of the graphical button selected to the control interface 204 of the processor 312, which determines the corresponding command message.

[0083] The user device 304 may also be used for recording audio and / or gestures using the application 344. In an example, the application 344 may include an option to enable a user to record audio to control the dialysis machine 200. After receiving selection of the option and recording the audio using a microphone on the user device 304, the application 344 converts the audio to text, a unique message, or signal using one or more speech-to-text algorithms (in some embodiments). The application 344 then transmits the text, unique message, or signal to the control interface 204 of the processor 312 for determining the corresponding audio command 307 using the data structure 314.

[0084] For gestures, selection of an option to input commends via gestures may cause the application 344 to activate a camera of the user device 304. The application 344 then transmits the recorded video to the gesture interface 208 of the processor 312. The gesture interface 208uses one or more algorithms to identify a hand gesture from the recorded video. In some embodiments, the algorithms may identify joints and other structural points on a user’s hand shown within the video. The algorithms create a mesh or wireframe model of the structural points and track movement using the model. The algorithms determine a gesture based on the tracked movement of the model. Gestures can include hand waving, hand rolling, a thumbs up sign, a pointer finger sign, clapping, gripping an object, and / or positioning fingers into the shape of a letter, symbol, and / or word.Touch, Audio, and / or Gesture Dialysis Machine Input Embodiment

[0085] Fig. 6 illustrates the dialysis machine 200 of Figs. 2 and 3 with the processor 312 configured to process touch, audio, and / or gesture inputs, according to an example embodiment of the present disclosure. For touch inputs, the dialysis machine 200 includes the display screen 306, which may include a touchscreen and / or physical buttons, knobs, etc. The dialysis machine 200 also includes the control interface 204.

[0086] In this embodiment, the control interface 204 selects which of the graphical user interfaces 314 are shown on the display screen 306. The control interface 204 receives a current operating state of the dialysis machine 200 from the control engine 210 and selects a corresponding graphical user interface 314 from the memory device 308. The control interface 204 transmits the selected graphical user interface 314 to a graphics processor 602, which is configured to render to the graphical user interface 314 on the display screen. The graphics processor 602 may populate data from the control engine 210 into data fields of the selected graphical user interface 314, such as status information or current operating parameters.

[0087] As discussed above in connection with Figs. 2 and 3 each graphical user interface 314 is defined by code that specifies coordinates of graphical buttons and other selectable features. The code also defines a function or an action that corresponds to selection of the button. When a user selects a graphical button of the graphical user interface 314, the touchscreen of the display screen transmits coordinates of the touch, which are used by the control interface 204 to determine which graphical button was selected and the corresponding action. The control interface 204 determines a touch command from the identified action using the data structure 314. The control interface 204 then transmits the touch command via the touch command message 205 to theresponse manager 202. When the action corresponds to a menu navigation action, the control interface 204 instead determines and causes the graphics processor 602 to renders for display on the display screen 306 the graphical user interface 314 that is specified by the action. Further, a “help” command may causes the control interface 204 to display a help graphical user interface 310. When an alarm is present and a “help” command is received, the control interface 204 may use information indicative of the alarm to determine corresponding help information, which may be displayed within a help graphical user interface 310. As such, the action may not necessarily correspond to an operation that is performed by the control engine 210.

[0088] For audio inputs, the dialysis machine includes at least one microphone 604. The microphone 604 may be located within the display screen 306. Alternatively, the microphone 604 may be located within a housing of the dialysis machine 200. In some embodiments, an array of microphones 604 may be positioned within different sections of the dialysis machine housing.

[0089] The microphone 604 is communicatively coupled to the processor 312. The microphone 604 is configured to transduce audio sound waves into analog and / or digital signals, which are transmitted to the audio interface 206 of the processor 312. When more than one microphone 604 is used, each microphone individually transmits analog and / or digital signals to the audio interface 206.

[0090] As discussed above, the audio interface 206 is configured to use one or more speech-to-text algorithms (in some embodiments) to convert the analog and / or digital signals to text, a unique message, or signal. The audio interface 206 then compares the text, unique message, or signal to libraries of key words and / or phrases that correspond to different commands. An audio command is selected based on the received text, unique message, or signal matching one or more key words and / or phrases that are associated with the command. In some instances, the audio interface 206 filters or selects the audio commands for keyword matching based on a current operating state of the dialysis machine 200, as communicated by the control engine 210. Additionally or alternatively, the audio interface 206 filters or selects the audio commands for keyword matching based on which of the graphical user interfaces 314 are displayed by the display screen 306. After an audio command is determined, the audio interface 206 transmits the audio command message 207 to the response manager 202 for selecting the corresponding dialysis machine command signal 211.

[0091] Fig. 7 is a diagram of an example process 700 performed by the audio interface 206 of Figs. 2 and 6 to determine an audio command from recorded audio, according to an example embodiment of the present disclosure. In the illustrated example, the process 700 begins when the microphone 604 transduces audio sound waves into analog and / or digital signals 702, which are transmitted to the audio interface 206. Next, the audio interface 206 uses one or more speech to text algorithms 703 to convert the analog and / or digital signals 702 to text, a unique message, or signal (block 704). The audio interface 206 then compares the text, unique message, or signal to one or more defined trigger words (block 706).

[0092] A trigger word is a single word or multiple words that indicate audio commands are being provided. The trigger word may include “Dialysis Machine” or “PrisMax 1”. Alternatively, the trigger word may be set by a user and can include virtually any word or group of words (e.g., “Derek”). Further, the trigger word does not necessarily need to be an English word. The use of a configurable trigger word prevents multiple dialysis machines 200 in the same location from acting upon an audio input after the trigger word is spoken. In some instances, the server 302 of Fig. 3 may determine which dialysis machines 200 are located in a same center or floor using, for example, an identifier and / or an Internet Protocol address of the dialysis machines 200. When the server 302 determines that the dialysis machines 200 are in close proximity of each other, the server 302 queries the dialysis machines 200 for their programmed trigger words. When the trigger words are identical or substantially close, the server 302 may transmit a trigger word change message to one or more of the dialysis machines 200, which causes the respective display screen 306 to display a prompt for a user to change the trigger word. In some instances, the dialysis machine 200 may prevent a user from starting a treatment until the trigger word is changed.

[0093] In some embodiments, the dialysis machine 200 may include more than one trigger word. For instance, in a dialysis center, hospital, or a patient’s home, multiple users may access the dialysis machine 200. A trigger word may be assigned for each user. As such, the dialysis machine 200 may log which user used the dialysis machine 200 based on the received trigger word.

[0094] Additionally or alternatively, the dialysis machine 200 may include voice recognition. In these examples, the audio interface 206 is additionally configured to determine if spoken audio includes an audio signature that matches an audio signature of an approved user. If the audio is not from an approved user, the audio interface 206 discards the audio and / or generatesan alert indicative of the unauthorized user. Such a configuration prevents unauthorized users from using their voice to change how the dialysis machine 200 is operating.

[0095] To enable audio commands to be provided at any time that the dialysis machine 200 is powered, the processor 312 may cause the microphone 604 to be actively recording. The processor 312 may cause the microphone 604 to turn off when an audio input setting is disabled, a patient requests that the audio input setting be disabled, and / or during certain operating conditions, such as software updates, during remote calls, or when the dialysis machine 200 is powered off. To prevent all of the recorded audio from being processed, the audio interface 206 uses the one or more trigger words. When text, a unique message, or signal is converted from recorded audio, the audio interface 206 compares the text, unique message, or signal to the one or more stored trigger words. When there is not a match, the audio interface 206 discards the received audio (block 708). This prevents unintended audio from being recorded, such as private conversations.

[0096] However, when the received text, unique message, or signal includes the trigger word, the audio interface 206 next determines if a command word is received within a time period from when the trigger word was recorded (block 710). As discussed above in connection with Figs. 3 to 5, the memory device 308 includes one or more data structures 314 that specify which words or phrases are commands (e.g., command words). A command word may include an action, such as “Pause” or “Pause treatment”. A command word may also include a data entry command, such as “enter a treatment time of two hours” or “confirm”. The time limit is used to ensure that audio is not recorded for a prolong period when a trigger word may have inadvertently been spoken or misinterpreted from other audio. The time limit may include two seconds, five seconds, ten seconds, etc. Further, in some embodiments, a button, light, a lightbar or icon on the display screen 306 may be illuminated, highlighted, or otherwise indicate when the audio input feature is activated on the dialysis machine 200 in response to detecting a trigger word. The button, light, lightbar, or icon may be dimmed or otherwise turned off after the time limit when a command word was not received (block 712). The audio interface 206 may further cause a sound to be emitted from a speaker of the dialysis machine 200 after the timeout.

[0097] As discussed above, the library of acceptable command words may be based on a current operating state of the dialysis machine 200 and / or which graphical user interface 310 isdisplayed. In these embodiments, the control engine 210 transmits a message 713 to the audio interface 206 that is indicative of a current operating state. The message 713 may indicate, for example, that the dialysis machine 200 is operating in a program mode, a treatment mode, a disinfection mode, a priming mode, a tubing setup mode, a post-treatment mode, a service mode, etc. Different modes may have different available command words based on the options for that mode. For example, the disinfection, priming, and treatment modes may each include command words such as “Start”, “Pause”, and “Resume” while the program mode includes command words for entering dialysis treatment parameter values into data fields to specify how the dialysis treatment is to be performed. Such command words may also be associated with the different graphical user interfaces 310 that are displayed.

[0098] In some embodiments, the command words may correspond to graphical buttons within the graphical user interface 310 and / or physical buttons on the dialysis machine 200. The audio interface 206 uses the data structure 314 to determine which commands correspond to graphical buttons within the graphical user interface 310 and / or physical buttons on the dialysis machine 200. Based on this identified association, the audio interface 206 may cause the buttons to illuminate, blink, or otherwise indicate which of the buttons are available as audio inputs.

[0099] Returning to Fig. 7, the process 700 also includes determining when the received text, unique message, or signal includes command parameter values (block 714). When the text, unique message, or signal includes a parameter value, the audio interface 206 appends the value to the selected command. For example, an audio input of “enter a treatment time of two hours” may be available for a program mode and / or when a treatment program graphical user interface 310 is displayed. The text of “treatment time” is identified by the audio interface 206 as the command word. Further, the audio interface 206 identifies the “two hours” as a parameter value that includes identified units (i.e., hours). The audio interface 206 determines that the treatment program graphical user interface 310 includes a data field corresponding to treatment time. After making the match, the audio interface 206 populates a value of two hours into the treatment time data field. Similar actions may be performed for audio inputs specifying a dextrose concentration of dialysis fluid, a dialysis fluid flow rate, a blood flow rate, a desired ultrafiltration removal volume and / or rate, a heparin infusion rate, a blood substitution infusion rate, etc. In another example, the graphical user interface 310 may prompt a user to confirm when dialysis tube isconnected to a patient’s catheter, a supply bag, and a drain bag. In this example, a parameter value is not available such that the data structure 314 instructs the audio interface 206 to search text for the word “confirm” or similar words, such as “ok”, “enter”, “accept”, etc. Each command within the data structure 314 may include one or more similar words that are indicative of the command. Further, the data structure 314 may include words with similar meanings for different languages.

[0100] As shown in Fig. 7, the process 700 also includes the audio interface 206 confirming that an audio command is enabled (block 716). Alternatively, the response manager 202 may determine if the command is enabled. In addition to using the current operating state message 713 to determine available command words, the current operating state may be used to determine if the audio command is enabled. For example, an audio command of “Stop” may be available for priming and disinfection modes, but may be disabled for a treatment mode since stopping or ending (different from pausing) a treatment is a more critical action that can only be performed through physical interaction with the dialysis machine 200. The data structure 314 specifies which of the audio commands are enabled based on the operating state of the dialysis machine 200.

[0101] In some embodiments, the audio command may be disabled based on a current operating condition of the dialysis machine 200. For instance, the data structure 314 may specify that all audio commands are disabled when the current operating condition includes an alarm. Thus, the audio interface 206 may disable audio inputs when the control engine 210 transmits a message indicative of a current alarm. Such a configuration prevents a patient (or other user) from silencing an alarm without viewing the display screen 306 of the dialysis machine 200. In other instances, audio inputs may be disabled when a speaker of the dialysis machine 200 is being used or is otherwise active.

[0102] In some embodiments, steps 710 and 716 may be combined such that the command is only present in the data structure 314 when the command is enabled for a given operating state and / or graphical user interface 310. The absence of a command for a certain operating state and / or graphical user interface 310 indicates, in this embodiment, that the command is not enabled. Thus, the audio interface 206 is configured to only respond to audio that corresponds to commands specified within the data structure for a given operating state of the dialysis machine 200 and / or displayed graphical user interface 310.

[0103] As shown in Fig. 7, when an audio command is not enabled, the audio interface 206 indicates an invalid entry (block 718). To provide this indication, the audio interface 206 may display an error message on the display screen 306. Additionally or alternatively, the audio interface 206 may cause a speaker to emit a sound or audible warning indicating the command is not enabled or otherwise invalid.

[0104] When the audio command is enabled, the audio interface 206 is configured to transmit the command via the audio command message 207 to the response manager 202. As shown in Fig. 7, the process 700 continues when the response manager 202 determines the dialysis machine command signal 211 that corresponds to the command (block 720). In some embodiments, the response manager 202 uses the data structure 314 to determine the corresponding dialysis machine command signal 211. Alternatively, the response manager 202 may use a different data structure that correlates each of the touch, audio, and / or gesture commands with corresponding dialysis machine command signals 211.

[0105] When the audio command is enabled and processed, the audio interface 206, in some embodiments, is configured to blink, illuminate, or otherwise indicate which graphical button and / or physical button on the dialysis machine 200 was selected through the audio / voice input. For example, an audio input for pausing a dialysis treatment may be entered by a user. After confirming the command is enabled, the audio interface 206 causes the pause button on the dialysis machine 200 to blink one to four times to provide a visual indication of the operation selected. The audio interface 206 may cause graphical buttons to blink, flash, or change color in a similar manner.

[0106] The example process 700 ends when the response manager 202 transmits the dialysis machine command signal 211 to the control engine 210. After receiving the dialysis machine command signal 211, the control engine 210 executes the command to change, begin, and / or end an operation of the dialysis machine 200. As shown in Fig. 6, the dialysis machine 200 may include a number of pumps and / or valves. For example, pumps may be used to circulate dialysis fluid within a dialysis fluid loop, which includes a dialyzer. Pumps may also be used to pump fresh dialysis fluid from a dialysis fluid source and / or pump used dialysis fluid to a drain. Further, pumps may be used to circulate a patient’s blood through an extracorporeal circuit, pump heparin into the extracorporeal circuit, and / or a replacement fluid into the extracorporeal circuit.Further, valves may be used to restrict fluid and / or blood flow. Some valves may also control pneumatic pressure for controlling the pumps. After the dialysis machine 200 changes operation based on the received audio / voice input, the example process 700 ends until a next audio trigger word is detected.

[0107] When the dialysis machine command signal 211 corresponds to navigation among the graphical user interfaces 210, the response manager 202 instead transmits the dialysis machine command signal 211 to the control interface 204. After receiving the dialysis machine command signal 211, the control interface 204 updates the current graphical user interface 910 or selects a requested graphical user interface. In some embodiments, when the audio command provides for navigation among graphical user interfaces 910, the audio interface transmits the command message 207 to the control interface 204 instead of the dialysis machine command signal 211.

[0108] Returning to Fig. 6, the processor 312 includes one or more gesture sensors 608 for receiving gesture inputs. The gesture sensors 608 may include a video camera, an infrared sensor, a motion sensor, a capacitive sensor, etc. The gesture sensors 608 are configured to record hand and / or arm movements that are converted into text, a unique message, or signal for processing as inputs for the dialysis machine 200.

[0109] The gesture sensor 608 may be activated as along as the dialysis machine 200 is powered. In other instances, the processor 312 is configured to activate the gesture sensor 608 only when gesture inputs can be received. For example, the gesture sensor 608 may be deactivated during software updates, servicing, treatments, and when the dialysis machine 200 is powering off. In some embodiments, the dialysis machine 200 may include a physical button or one or more graphical user interfaces 910 include a graphical button, which when pressed, activates the gesture sensor 608 and / or the gesture interface 208.

[0110] Figs. 8A to 8C show example gestures, according to example embodiments of the present disclosure. It should be appreciated that the gestures are provided for illustrative purposes and can include virtually any hand and / or arm gesture. Gesture 810 of Fig. 8A includes a hand swiping motion. Gesture 820 of Fig. 8B includes a thumbs up motion. Gesture 830 of Fig. 8C includes a touch of a virtual interface.

[0111] When the gesture sensor 608 is a camera or proximity sensor, the sensor 608 records a stream of video images (or sequence of distances of arm / hand structure from the sensor) of a gesture, such as the gestures 810 and 820. The video images show a change in movement over time. The example gesture interface 208 is configured to use one or more image recognition algorithms to identify hand / arm orientation and positioning. In some instances, the image recognition algorithm analyzes the video images to identify arm and finger joints, which are designated as nodes. The algorithm then identifies bodily structure between the joints, which are designated as lines between the nodes. The algorithm accordingly creates a wireframe or grid model of a user’s hand / arm. The algorithm next overlays the wireframe or grid model over the images, aligning the nodes at the joints and rotating the lines at the nodes to coincide with the detected movement. The gesture interface 208 records the movement of the wireframe or grid model through the sequence of images. The gesture interface 208 then compares the recorded movement to a library of possible movements to determine a match. When there is a match, the gesture interface 208 is configured to output text that corresponds to the movement, such as “thumbs up”, “swiping motion”, “waving motion”, etc.

[0112] When the gesture sensor 608 as an infrared sensor, the gesture interface 208 receives heat signatures. A single sensor may be used to detect when a user’s hand or finger comes in proximity to the gesture sensor 608. An array of sensors may be used to track movement of the hand and / or arm. In this embodiment, the gesture sensor 608 is configured to use the IR sensor data over time to detect movement and / or proximity triggers. In response, the gesture sensor 607 is configured to out text, a unique message, or signal corresponding to the detection, such as “selection” or “hand wave”. The gesture sensor 608 may use one or more algorithms for detecting the arm / hand / fingers within the infrared data and identifying movement over a time-series analysis of the data. The infrared gesture sensor 608 may be used to detect the gesture 830 shown in Fig. 8C, which represents a user’s finger contacting the virtual plane, which has a known proximity to the sensor.

[0113] When the gesture sensor 608 is a capacitive sensor, the gesture interface 208 receives capacitance data, which is indicative of a change in capacitance over time. The capacitive gesture sensor 608 may be used for detecting proximity of a user’ s finger, for example, from a sensor or metallic surface of the dialysis machine 200. In some embodiments, the gesturesensor 208 includes the touchscreen of the display screen 306. The capacitive gesture sensor 608 may be used to detect the gesture 830 shown in Fig. 8C, which represents a user’s finger contacting the virtual plane, which has a known proximity to the sensor. Similar to the other gesture sensors 608, the gesture interface 608 outputs text, a unique message, or signal corresponding to the identified gesture that was detected using capacitance.

[0114] Similar to the audio interface 206, the gesture interface 208 is configured to use the data structure 314 to match the text, unique message, or signal indicative of the gesture to a command. The match may be based on a current operating state of the dialysis machine 200 and / or which graphical user interface 310 is displayed. For example, the display screen 306 may display a graphical user interface 910 prompting a user to confirm dialysis tubing is connected. The graphical user interface 910 includes a confirm graphical button. The data structure 314 associates words corresponding to “confirm” as commands. After detecting a thumbs up gesture via the gesture sensor 608, the gesture interface 208 outputs text, a unique message, or signal that includes “acceptance”. The data structure 314 includes the term “acceptance” as an alternative of “confirm” and generates the corresponding gesture command as an input to indicate the dialysis tubing was connected. The gesture interface 208 transmits the command message 209 to the response manager 202 indicative of the confirmation, which then uses the data structure 314 (or a different data structure) to determine the dialysis machine command signal 211 that corresponds to the confirmation gesture command, which is then transmitted to the control engine 210.

[0115] The gesture interface 208 may also ensure that a command is enabled before generating the gesture command message 209. Alternatively, the response manager 202 is configured to determine whether a command is enabled. As discussed above, the data structure 314 indicates when certain gesture commands are enabled and disabled based, for example, on an operating state of the dialysis machine 200 and / or which graphical user interface 310 is displayed on the display screen 306. As part of identifying a command from text, a unique message, or signal describing a gesture, the gesture interface 208 (or response manager 202) determines whether the gesture input is enabled. When a gesture is disabled, the gesture interface 208 (or response manager 202) discards the gesture. In some instances, the gesture interface 208 (or response manager 202) generates a visual alert for display on the display screen and / or an audio alert indicative that the gesture input was not accepted. Further, when a gesture cannot be matched toavailable commands for a current operating state of the dialysis machine and / or the currently displayed graphical user interface 314, the gesture interface 208 (or response manager 202) may be configured to generate an alert.Additional Touch, Audio, and / or Gesture Dialysis Machine Input Embodiment

[0116] Fig. 9 is an alternative embodiment of how touch, audio, and / or gesture inputs may be processed within the dialysis machine of Fig. 3, according to an example embodiment of the present disclosure. In the illustrated embodiment, the response manager 202 of the dialysis machine 200 is configured to receive touch inputs 902 from the control interface 204, audio inputs 904 from the audio interface 206, and gesture inputs 906 from the gesture interface 208. The touch inputs 902 may include touchscreen coordinates and / or an indication of a physical button that was pressed. The audio inputs 904 include text, a unique message, or signal that was converted from analog and / or digital signals that were recorded of sound waves. The gesture inputs 906 include text, a unique message, or signal that describes a gesture recorded by the one or more gesture sensors 608. As such, the interfaces 202, 206, and 208 transmit signals and / or messages that are indicative respectively of touch, audio, and / or gesture inputs.

[0117] The response manager 202 is configured to use the one or more data structures 314 of Figs. 4 and 5 to determine the corresponding touch commands, audio commands, and gesture commands from the respective inputs 902, 904, and 906. For example, for audio inputs 904, the response manager 202 is configured to the use the data structure 314 to determine when the text, unique message, or signal includes a command word. When the text, unique message, or signal includes a command word, the response manager 202 identifies the corresponding command using, in part, a current operating state of the dialysis machine 200 and / or an indication of which graphical user interface 310 is displayed. The response manager 202 also uses the current operating state of the dialysis machine 200 and / or an indication of which graphical user interface 310 is displayed to determine if the audio command is enabled. When the audio command is enabled, the response manager 202 transmits the corresponding audio command message 207 to a user interface response module 908, which then uses the data structure 314 or a different data structure to determine a corresponding dialysis machine command signal 211. The user interfaceresponse module 908 then transmits the command signal 211 to the control engine 210 to change an operation of the dialysis machine 200.

[0118] The response manager 202 may perform similar operations for the gesture input 906 and the touch input 902. The user interface response module 908 provides a further layer between the control engine 210 and the interfaces 204, 206, and 208 that enables conversion steps to be further partitioned. For instance, the interfaces 204 to 208 only transmit messages indicative of inputs rather than commands. The response manager 202 instead determined the commands from the inputs 902 to 906. Further, the user interface response module 908 determines the dialysis machine command signal 211 rather than the response manager 202. The addition of the user interface response module 908 may enable the different input modalities to be more easily added and removed to the front end architecture of the dialysis machine 200 without affecting the other modalities or the operating structure of the control engine 210.I. Conclusion

[0119] It should be understood that various changes and modifications to the presently preferred embodiments described herein will be apparent to those skilled in the art. Such changes and modifications can be made without departing from the spirit and scope of the present subject matter and without diminishing its intended advantages. It is therefore intended that such changes and modifications be covered by the appended claims.

Claims

CLAIMSThe invention is claimed as follows:Claim 1 : A dialysis machine comprising: a memory device storing audio rules, the audio rules specifying:(i) which audio commands are enabled based on a current state of the dialysis machine, and(ii) which dialysis machine command signal is to be selected based on an audio command and the current state of the dialysis machine; a display screen including buttons for entering commands; a control interface configured to render graphical user interfaces for display by the display screen and output a touch command indicative of a specified dialysis machine operation to be performed; a microphone; an audio interface configured to convert audio signals from the microphone into an audio command indicative of a specified dialysis machine operation to be performed; a control engine configured to perform dialysis machine operations based on received dialysis apparatus command signals; and a processor communicatively coupled to the memory device, the display screen, the control interface, the control engine, and the audio interface, the processor including: a response manager configured to: receive one of (a) the touch command from the control interface, or (b) the audio command from the audio interface, communicate with the control engine to determine a current state of the dialysis machine, when the audio command is received and the (i) audio rules indicate the audio command is enabled, determine the dialysis machine command signal that corresponds to the audio command using the (ii) rules and the current state of the dialysis machine, and transmit the dialysis machine command signal to the control engine.Claim 2: The dialysis machine of Claim 1, wherein the current state of the dialysis machine includes at least one of programming setup, fluid line setup, disinfection, fluid line priming, recirculation, blood return, treatment administration, and a service mode.Claim 3: The dialysis machine of Claim 1, wherein the current state includes an indication of the graphical user interface displayed on the display screen.Claim 4: The dialysis machine of Claim 1, wherein the control interface includes at least one of buttons, knobs, switches, control surfaces, dials, or a touchscreen integrated with the display screen.Claim 5: The dialysis machine of Claim 1, wherein the dialysis machine command signals specified by the audio rules are indicative of at least one of setting or changing a dialysis machine treatment parameter, starting a dialysis treatment, pausing a dialysis treatment, stopping a dialysis treatment, muting or silencing an alarm or alert, starting a priming operation, stopping a priming operation, acknowledging a treatment setup step is complete, starting a disinfection operation, initiating a call to a clinician, changing a language of the audio commands, changing a trigger word, selecting a feature or option that is provided within a user interface shown on the display screen, navigating between different user interfaces, powering off the display screen, or begin recording audio.Claim 6: The dialysis machine of Claim 1, wherein the audio command includes at least one of a command word, a command phrase, or a command word and parameter values.Claim 7: The dialysis machine of Claim 1, wherein the audio interface is configured to convert the audio signals from the microphone into the audio command by: converting the audio signals into text, a unique message, or signal; comparing the text, unique message, or signal to the audio commands provided in the audio rules; and when there is a match or favorable comparison between the text, unique message, or signal and one of the audio commands, transmitting the matched or favorably comparing audio command to the response manager.Claim 8: The dialysis machine of Claim 7, wherein the audio rules additionally include at least one trigger word, and wherein the audio interface is further configured to: compare the text, unique message, or signal to the at least one trigger word, and when there is a match between the text, unique message, or signal and the at least one trigger word, compare the text, unique message, or signal or additional text, unique messages, or signals to the audio commands provided in the audio rules for converting the audio signals from the microphone into the audio command.Claim 9: The dialysis machine of Claim 8, wherein the audio interface is further configured to: start a timer when the at least one trigger word is detected within the text; and compare the text, unique message, or signal or the additional text, unique messages, or signals to the audio commands until the timer reaches a predefined time limit.Claim 10: The dialysis machine of Claim 8, wherein the response manager is further configured to, when the audio command is received: determine which button or option of the control interface corresponds to the audio command; and illuminate or otherwise highlight the button or option of the control interface for a specified duration.Claim 11 : The dialysis machine of Claim 1, wherein the dialysis machine includes at least one of a hemodialysis machine, a hemodiafiltration machine, a peritoneal dialysis machine, or a continuous renal replacement therapy (“CRRT”) machine.Claim 12: The dialysis machine of Claim 1, wherein when the response manager receives the touch command from the control interface, the response manager is configured to: determine the dialysis machine command signal that corresponds to the touch command using the (ii) rules; and transmit the dialysis machine command signal to the control engine.Claim 13: The dialysis machine of Claim 1, wherein when the audio command is received and the (i) audio rules indicate the audio command is disabled, the response manager is configured to at least one of: discard the audio command; or generate an alert indicative of the audio command being discarded.Claim 14: The dialysis machine of Claim 1, wherein at least one of the control interface and the audio interface are included within the processor.Claim 15: The dialysis machine of Claim 1, further comprising: a gesture sensor; and a gesture interface communicatively coupled to the gesture sensor, the gesture interface configured to convert video or data indicative of arm / hand movements into a gesture command that is indicative of a specified dialysis machine operation to be performed.Claim 16: The dialysis machine of Claim 15, wherein the response manager is further configured to: receive the gesture command from the gesture interface; when the gesture command is received and gesture rules indicate the gesture command is enabled, determine the dialysis machine command signal that corresponds to the gesture command using the gesture rules and the current state of the dialysis machine; and transmit the dialysis machine command signal to the control engine.Claim 17: The dialysis machine of Claim 1, further comprising, at least one pump, pinch valve, or solenoid configured to control a flow of fluid to perform at least some of the dialysis machine operations, wherein the control engine is configured to control the at least one pump, pinch valve, or solenoid based on the dialysis apparatus command signal.Claim 18: A method comprising: receiving, in an audio interface of a dialysis machine from a microphone of the dialysis machine, an audio input; converting, via the audio interface, the audio input into text, a unique message, or signal; determining, via the audio interface, whether the received text, unique message, or signal includes a specified trigger word; when the received text, unique message, or signal includes the specified trigger word, determining, via the audio interface, if a command word is included within text, unique message, or signal that is received within a time period from when the trigger word was received; when the command word is received within the time period, using, via the audio interface, a data structure of audio rules and a current operating state of the dialysis machine to determine if an audio command corresponding to the command word is enabled; transmitting an audio command message indicative of the audio command to a response manager of the dialysis machine when the command word is enabled; determining, via the response manager using the same data structure or a different data structure, a dialysis machine command signal that corresponds to the audio command; andtransmitting from the response manager to a control engine of the dialysis machine, the dialysis machine command signal.Claim 19: The method of Claim 18, further comprising when a command word is not received during the time period, discarding, via the audio interface, additionally received text, unique messages, or signals until the trigger word is received again.Claim 20: The method of Claim 18, further comprising causing at least one button or graphical button of the dialysis machine to illuminate or blink when the when the command word is enabled.Claim 21 : The method of Claim 18, further comprising, when the audio command is not enabled, discarding the command word and / or causing a message to be displayed indicative of the audio command being disabled.Claim 22: A dialysis machine comprising: a memory device storing gesture rules, the gesture rules specifying:(i) which gesture commands are enabled based on a current state of the dialysis machine, and(ii) which dialysis machine command signal is to be selected based on a gesture command and the current state of the dialysis machine; a display screen including buttons for entering commands; a control interface configured to render graphical user interfaces for display by the display screen and output a touch command indicative of a specified dialysis machine operation to be performed; a gesture sensor; an gesture interface communicatively coupled to the gesture sensor, the gesture interface configured to convert video or data indicative of arm / hand movements into a gesture command that is indicative of a specified dialysis machine operation to be performed;a control engine configured to perform dialysis machine operations based on received dialysis apparatus command signals; and a processor communicatively coupled to the memory device, the display screen, the control interface, the control engine, and the gesture interface, the processor including: a response manager configured to: receive one of (a) the touch command from the control interface, or (b) the gesture command from the gesture interface, communicate with the control engine to determine a current state of the dialysis machine, when the gesture command is received and the (i) gesture rules indicate the gesture command is enabled, determine the dialysis machine command signal that corresponds to the gesture command using the (ii) rules and the current state of the dialysis machine, and transmit the dialysis machine command signal to the control engine.Claim 23: The dialysis machine of Claim 22, wherein the gesture sensor includes at least one of a camera, an infrared sensor, a capacitive sensor, or a depth sensor.Claim 24: The dialysis machine of Claim 22, wherein when the gesture command is received and the (i) gesture rules indicate the gesture command is disabled, the response manager is configured to at least one of: discard the gesture command; or generate an alert indicative of the gesture command being discarded.Claim 25: The dialysis machine of Claim 22, further comprising: a microphone; and an audio interface electrically coupled to the microphone, the audio interface configured toonvert audio signals from the microphone into an audio command indicative of a specifiedialysis machine operation to be performed, wherein the response manager is further configured to: receive the audio command from the audio interface, when the audio command is received and audio rules indicate the audio command is enabled, determine the dialysis machine command signal that corresponds to the audio command using the audio rules and the current state of the dialysis machine, and transmit the dialysis machine command signal to the control engine.

Images