Contactless input ultrasonic control

Abstract

A method and apparatus for controlling an ultrasound machine using one or more non-contact inputs is disclosed herein. In one embodiment, the method for controlling the operation of an ultrasound machine includes: obtaining one or more non-contact inputs; determining one or more operations for controlling the ultrasound machine based on the one or more non-contact inputs and a machine state of the ultrasound machine; and controlling the ultrasound machine using the one or more operations.

Description

Technical Field

[0001] One or more exemplary embodiments relate to an ultrasound machine and method of operating the same, and more particularly, to an ultrasound machine that uses lip reading to at least partially generate commands to control the operation of the ultrasound machine. Background Art

[0002] The ultrasound system radiates the ultrasound signal generated from the ultrasound probe into an object such as a patient, and receives the echo signal reflected from the inside of the object. An image of the inside of the object is generated using the received echo signal. More specifically, the ultrasound diagnostic machine generates an ultrasound image by using the ultrasound image data acquired from the ultrasound probe, and displays the generated ultrasound image on a screen to provide the ultrasound image to a user. The ultrasound machine may include a control panel for controlling the ultrasound machine and setting various functions.

[0003] Generally, an ultrasound machine has a plurality of function keys for receiving user input and an input device such as a keyboard as part of a control panel. In order for a user to control an ultrasound system including an ultrasound probe, various input units on the control panel must be manipulated by the user, which causes inconvenience when using the ultrasound system. In particular, when a clinician uses an ultrasound probe to perform a diagnosis of a patient, it may be inconvenient for the user to manipulate the control panel that is spaced apart from the user and requires a long time to manipulate.

[0004] In addition, clinicians often need to adjust the ultrasound machine during the sterile process. However, since clinicians work in a sterile field, but the ultrasound machine is not sterile, touching the ultrasound machine will destroy the sterility, so it is not easy for clinicians to do this. Moreover, both hands are often at work, one hand holding the probe and the other hand holding an instrument such as an injection or biopsy needle, so there is no extra hand to reach for the machine controls. Clinicians often solve this problem by having a nurse or assistant adjust the machine for them, but this is inefficient and not always feasible. Clinicians sometimes use sterile instruments such as cotton swabs to adjust the ultrasound machine, but this is inconvenient and requires handling the cotton swabs for each adjustment (because dirty cotton swabs should not be returned to the sterile field).

[0005] Some ultrasound machines use voice control of the ultrasound machine to address this issue. Typically, this doesn’t work well because hospitals tend to be very noisy spaces and it’s difficult for voice control to pick out commands from background noise or other conversations. This is especially important in a hospital setting because users absolutely don’t want the ultrasound machine to change state without being specifically commanded, especially at critical moments in a procedure.

[0006] Some ultrasound machines have buttons at the probe to control the ultrasound machine. These buttons may be inconvenient to use due to the different grip positions required when the buttons are present and the probe may be covered by a sterile sheath. Summary of the invention

[0007] A method and apparatus for controlling an ultrasound machine using one or more non-contact inputs is disclosed herein. In an embodiment, the method for controlling the operation of an ultrasound machine includes: obtaining one or more non-contact inputs; determining one or more operations for controlling the ultrasound machine based on the one or more non-contact inputs and a machine state of the ultrasound machine; and controlling the ultrasound machine using at least one of the one or more operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The present invention can be more fully understood through the detailed description given below and the drawings of various embodiments of the present invention. However, these drawings should not be considered to limit the present invention to specific embodiments, but are only for description and understanding.

[0009] Figure 1 is a flow chart of one embodiment of a process for controlling an ultrasound machine using contactless input.

[0010] Figure 2 is a flow chart of one embodiment of a process for triggering lip reading.

[0011] Figure 3 is a block diagram of one embodiment of an ultrasound system having a command generator.

[0012] Figure 4 A block diagram of one embodiment of a recognizer having one or more recognition components for executing one or more recognition procedures is shown.

[0013] Figure 5 is a block diagram of an embodiment of an ultrasound system with a command generator that includes artificial intelligence or machine learning.

[0014] Figure 6 is a flow chart of one embodiment of a process for controlling an ultrasound machine using a neural network. DETAILED DESCRIPTION

[0015] In the following description, many specific details are set forth to provide a more thorough description of the present invention. However, it will be apparent to those skilled in the art that the present invention can be implemented without these specific details. In other cases, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the present invention.

[0016] Methods and apparatus for generating commands to control an ultrasound machine using lip reading and machine state information of an ultrasound machine are disclosed. Using lip reading to control an ultrasound machine can achieve hands-free operation, which is very advantageous, particularly in a sterile medical environment. In one embodiment, non-contact commands are used to determine the commands. Video images and / or audio can be used to identify non-touch commands. In one embodiment, the non-contact commands include a combination of lip reading and additional information, such as, but not limited to: one or more types of examinations performed by the ultrasound machine, characteristics of the ultrasound image, user expectations of what the ultrasound machine might do next, etc. In one embodiment, the commands are determined using a command generator that includes a neural network (e.g., a deep learning neural network) or other artificial intelligence functionality.

[0017] Figure 1 1 is a flow chart of one embodiment of a process for controlling an ultrasound machine using non-contact input. The process is performed by processing logic that may include hardware (circuitry, dedicated logic, etc.), software (such as that running on a general purpose computer system or a dedicated machine), firmware, or a combination of the three.

[0018] Reference Figure 1 The process begins by performing lip reading, which includes capturing lip movements through one or more cameras and performing lip recognition on these lip movements (processing block 101). There are various techniques for performing lip reading by capturing lip images, analyzing the images (e.g., measuring the height and width of the lips and other features such as the shape of an ellipse surrounding the lips based on the images captured by the cameras) to determine the movements of the lips, identifying a series of shapes formed by the mouth, and then matching them to specific words or word sequences. In one embodiment, lip reading is performed using a neural network (e.g., a deep learning neural network).

[0019] Next, processing logic determines an operation for controlling the ultrasound machine based on the one or more non-contact inputs and the machine state information of the ultrasound machine (processing block 102). In one embodiment, the operation is determined using a command generator of the ultrasound machine. In one embodiment, the command generator is implemented as processing logic, which may include hardware (circuitry, dedicated logic, etc.), software (e.g., software running on a general purpose computer system or a dedicated machine), firmware, or a combination of the three.

[0020] In one embodiment, the command generator receives various inputs and determines commands based on these inputs. In one embodiment, the command generator includes a recognizer or recognition engine that performs one or more recognition procedures on the input data to generate a recognition result based on which the command that the user wants or will execute is determined. In one embodiment, the command generator uses a neural network (e.g., a deep learning neural network) as part of the process of determining the operation to be performed by the ultrasound machine. One such embodiment is described in more detail below.

[0021] In one embodiment, the generated operations include commands for controlling operating parameters, such as, but not limited to, adjusting gain (e.g., increasing gain, decreasing gain) and adjusting depth (e.g., increasing depth, decreasing depth). Other operating parameters may also be generated, such as transducer selection, on / off mode (e.g., A-mode, B-mode or 2D mode, B-flow, C-mode, M-mode, Doppler mode (e.g., color Doppler, continuous wave (CW) Doppler, pulsed wave (PW) Doppler, duplex, triplex, etc.), pulse inversion mode, harmonic mode, etc.

[0022] In one embodiment, the operations include freezing an ultrasound image displayed by the ultrasound machine, saving an image displayed by the ultrasound machine, adding and / or moving annotations (e.g., labels) or pictograms on the image displayed by the ultrasound machine, and creating or filling out a report (e.g., a billing record, a medical report, etc.) using information stored on or received by the ultrasound machine.

[0023] In one embodiment, the one or more contactless inputs include the results of lip reading recognition, captured audio information, image information of an ultrasound image displayed by the ultrasound machine, the type of examination performed by the individual using the ultrasound machine, a list of operations performed by the user of the ultrasound machine (e.g., a workflow), and / or a next action to be performed by the user of the ultrasound machine. In one embodiment, one or more such inputs are included in the state information fed back from the ultrasound control subsystem of the ultrasound machine for command generation.

[0024] In one embodiment, the ultrasound system uses lip reading recognition results combined with voice data to improve the accuracy of control. In one embodiment, audio (e.g., the user's voice) is captured with a microphone and recognized for use in conjunction with lip reading recognition results in determining an operating parameter to be changed or another operation the user wants to perform. In one embodiment, the audio is spoken by a user of the ultrasound machine. Thus, in this manner, both the audio and image recognition associated with the user speaking the command are used to determine an operating parameter to be adjusted or another operation the user wants to perform.

[0025] In one embodiment, the influence of the selection of inputs to be used (e.g., contactless inputs) or all of these inputs on the command determination changes based on one or more factors. These factors can be environmental factors or other information obtained about the environment (e.g., information sensed from the environment). For example, in one embodiment, the outputs from lip reading recognition and audio recognition are weighted so that each does not have the same influence or contribution to the determination of the operation identified based on information about the environment. In one embodiment, based on the noise in the environment, the system determines the operation that the user wants to perform based more on the results of the lip reading operation than the results of the voice recognition, and vice versa. In one embodiment, this is achieved by dynamically adjusting the weights of the outputs from lip reading recognition and audio recognition. Therefore, if the environment is determined to be too noisy (e.g., the noise level is above a threshold), the results of the lip reading recognition can be weighted higher than the results of the audio recognition (or only the results of the lip reading recognition) when determining the operation that the user wants to perform.

[0026] In one embodiment, the ultrasound system ignores contactless input to the ultrasound system until the ultrasound system enters an activation mode when it is activated to receive contactless input. Before the ultrasound system enters the activation mode, the ultrasound system determines that the user is looking directly at the camera. In this case, the ultrasound system ignores any speech unless the user is looking directly at the camera when speaking. Since clinicians typically look at the machine when making command-based adjustments, this does not impose any practical limitations on clinicians. In another embodiment, the activation mode of the ultrasound system requires a gesture such as a facial prompt to start listening to speech and / or studying visual data for lip reading. Examples of such facial prompts include, but are not limited to, closing the eyes for a certain predetermined period of time (i.e., a long blink), blinking, nodding, etc.

[0027] It should be noted that the lip reading recognition program may use other data such as the exam type, work list, system login information, or other status information stored in and available in the ultrasound machine to determine the operation that the user wants to perform. Similarly, image information associated with the image displayed by the ultrasound machine may also be used by the lip reading recognition program to determine the operation that the user wants to perform.

[0028] After determining one or more operations based on the results of the lip reading recognition and the machine state of the ultrasound machine, in one embodiment, the processing logic generates one or more user-selectable operations that can be performed by the ultrasound machine and causes them to be displayed (processing block 103). This will initiate a confirmation mode process, enabling the user to select and / or confirm the operation to be performed by the ultrasound machine. This is particularly advantageous in situations where the command generator is uncertain about what operation the user wants to perform. This uncertainty may be caused by the following reasons: the lip reading recognition result has a confidence level of less than 100% in its determination, some other recognition (e.g., audio recognition) result has a confidence level of less than 100% in its determination, and / or some other limitation on the accuracy of the input used in the command generation process (e.g., a mismatch between audio-based commands and lip reading-based commands, etc.).

[0029] In one embodiment, the selectable operations are presented on a display and / or using audio (e.g., computer-generated speech) of the ultrasound machine under control of the ultrasound imaging subsystem of the ultrasound machine. In one embodiment, the selectable operations are presented in a list based on their likelihood (e.g., confidence) of matching the user's desired operation as determined by a command generator of the ultrasound machine. In one embodiment, the list of selectable operations includes a confidence factor generated by the command generator for each operation, or other information for providing the user with an indication of the confidence level associated with the operation determined by the ultrasound machine.

[0030] It should be noted that presenting and / or confirming an operation is optional and not required. In an alternative embodiment, a single operation may be determined and a command executed by the ultrasound machine may be generated without selection and / or confirmation.

[0031] Next, in one embodiment, processing logic uses the operations generated based on the lip reading and control state information to control the ultrasound machine (processing block 104). In another embodiment, the operations are also generated based on the audio information. In one embodiment, control is implemented in response to selection and / or confirmation of the operations, as necessary. In one embodiment, control is performed by a control subsystem of the ultrasound machine.

[0032] As described above, the determination of the command is based on performing lip reading and recognition of lip movements. In one embodiment, the lip reading process is triggered in response to an activation mode, wherein the activation mode involves the occurrence of one or more operations. In one embodiment, one of the operations is to determine that the user is looking at the camera when speaking or lipping a command to be executed by the ultrasound machine. In another embodiment, lip movement recognition occurs in response to the user performing a gesture that is recognized by the ultrasound machine. In one embodiment, the gesture includes performing a facial prompt. In one embodiment, the facial prompt includes the user closing their eyes for a predetermined period of time, the user blinking, the user nodding, or any other facial gesture. Figure 2 is a flow chart of one embodiment of a process for triggering the lip reading performed above. In an embodiment, the process is performed by a recognizer or recognition engine of a command generator of an ultrasound machine.

[0033] Reference Figure 2 The process begins by determining a user who issues a contactless command (processing block 201). In one or more embodiments, the determination is made by at least one of: determining that the user is looking at (e.g., directly looking at) a camera, determining that the user has performed a gesture, and using a user identification or authentication process.

[0034] Determining that a user is looking at (e.g., directly looking at) a camera or that a user has performed a gesture requires image data from a camera. Such a camera may be mounted on or embedded in the system, located on a probe of an ultrasound system, or attached to an ultrasound system. In one embodiment, the ultrasound machine uses eye tracking to determine that the user is looking at the camera. Eye tracking uses a camera to record eye movements and processes these images to determine the location of the user's gaze in a manner well known in the art. In one embodiment, the camera used to perform eye tracking is the same camera used to capture lip movements for lip reading, but a separate camera may also be used. In one embodiment, eye tracking is enhanced so that the system can determine individuals that need to be tracked in the inspection area. In one embodiment, this enhancement includes determining the orientation of the face, facial recognition based on the person starting the inspection, and / or accessing information from a worklist indicating the sonographer, physician, or other medical worker who will be using the ultrasound machine for the inspection. It should be noted that more than one user of an ultrasound machine for the same inspection can be identified in this manner.

[0035] In one embodiment, if the identified physician is a neonatologist, the settings of the ultrasound system are optimized to visualize smaller anatomical structures commonly seen in a typical neonatal exam. In one embodiment, after the exam begins, the physician's identity is used as part of a machine learning process to more accurately predict settings that the clinician or user may attempt to adjust. A profile associated with the user login is also used to determine initial system settings.

[0036] In one embodiment, determining whether the user has performed a gesture that triggers the lip reading recognition process is based on facial prompts or other wake-up gestures performed by the user. Other wake-up gestures include but are not limited to the user closing their eyes for a predetermined period of time (i.e., long blinking), blinking, nodding, etc.

[0037] In response to determining that the user is looking at the camera or determining that the user has performed a gesture, processing logic performs lip reading (processing block 202) and the remainder of the command generation process as described herein.

[0038] In an alternative embodiment, the system performs facial recognition or some other user identification / authentication process on the individual and only responds to commands from the face of the individual initiating the examination. That is, the system identifies the individual from a group of individuals that may be in the vicinity of the system. This is accomplished using a user identification operation (e.g., facial recognition, obtaining ultrasound data such as a work list, or other information identifying the individual being examined, etc.).

[0039] After determining that the user is looking at (e.g., looking directly at) the camera, it is determined that the user has performed a gesture, and by using a user identification / authentication process, the system provides the identified individual with control of the ultrasound machine using the contactless input. In other words, the system allows the individual to provide control of the system using their contactless input.

[0040] In one embodiment, once the ultrasound system determines the individual(s) from whom it will accept contactless commands for a particular examination, the ultrasound system does not accept contactless commands from other individuals who may be present in the examination.

[0041] Figure 3 is a block diagram of one embodiment of an ultrasound system having a command generator as described above. In one embodiment, the ultrasound machine includes a transducer probe for sending and receiving sound waves (echoes) in a manner known in the art, which are processed to produce an ultrasound image. Figure 3 The transducer probe is not shown in order to avoid obscuring the technology disclosed herein.

[0042] Reference Figure 3 , the ultrasound control subsystem 332 includes one or more processors. One processor causes current to be sent to the transducer probe to transmit sound waves, and also receives electrical pulses generated by echoes from the probe. The processor processes the raw data associated with the received electrical pulses and forms an image. The processor sends the image data to the ultrasound imaging subsystem 332, and the ultrasound imaging subsystem 332 displays the image on the display 340. Therefore, the display screen 340 displays an ultrasound image of the ultrasound data processed by the processor of the ultrasound control subsystem 332.

[0043] Figure 3The ultrasound system also includes one or more cameras 301 to capture images or video information stored in the memory 304. In an alternative embodiment, the microphone 302 records audio information also stored in the memory 304. In one embodiment, the command generator 306 can receive one or more other non-audio inputs. In one embodiment, these other non-audio inputs include one or more of the following feedback: image information feedback 310 from the ultrasound imaging system 331 of the ultrasound machine, and machine state feedback 311 from the ultrasound control subsystem 332 of the ultrasound machine. In one embodiment, the other non-audio input data includes workflow information for a given user, or workflow information for a given type of procedure or examination, used to predict or assist in predicting the next command or step in the examination or process. These can also be stored in the memory 304. Although the memory 304 is Figure 3 301, microphone 302, and non-audio input 303 are stored in more than one memory. In another additional alternative embodiment, the data is transmitted directly to a processor in a cloud server or local server (e.g., cloud server 340) that performs the operations described herein via a network communication interface. In an alternative embodiment, the data is sent directly to a neural network (e.g., a deep learning neural network, etc.) for processing.

[0044] The command generator 306 accesses the memory 304 and, in response thereto, generates one or more operations (e.g., commands) 320 for controlling the ultrasound machine. In one embodiment, the command generator 306 is integrated into the ultrasound machine. In another embodiment, the command generator 306 is a standalone device that can be connected to the ultrasound machine via a wired and / or wireless connection. In another embodiment, the command generator 306 is part of a cloud-based computer resource that can be connected to the ultrasound machine via a wired and / or wireless connection.

[0045] In one embodiment, the command generator 306 includes one or more processors, neural networks (e.g., deep learning neural networks, etc.), etc., to control the operation of the ultrasound system by generating control options or operations. In one embodiment, the command generator 306 includes a recognizer or recognition engine 305, which, as part of the generating operation, performs recognition on the image data and audio data captured by the camera 301 and the microphone 302, respectively. For example, the recognizer 305 accesses the memory 304 to obtain the captured image information from one or more cameras 301, and performs lip reading recognition on the lip movements captured by the camera 301 to determine the specific command from the individual who is using the ultrasound machine. For example, the user may have lip-read a command to increase the gain on the image being displayed by the ultrasound imaging subsystem 331. The command is recognized by the recognizer 305, and in response thereto, the command generator 306 accesses the current gain value provided in the machine state information 311, and generates a command to increase the gain of the ultrasound machine from its current value to the increased value. In one embodiment, the gain increase is a preset amount. In another embodiment, the command provided by the user includes a desired parameter change amount of "increase the gain by 10%". In another embodiment, the gain increase that occurs when the command is executed is specified by the user and recognized by the recognizer 305. In another embodiment, as the network machine learns image recognition features, the optimal increment is determined and the correct amount of gain is applied so that the displayed ultrasound image is optimized. After the command is generated, the command is sent to the ultrasound control subsystem 332, which controls the ultrasound imaging subsystem 331 and increases the gain of the image being displayed by the ultrasound imaging subsystem 331. In one embodiment, the same process is performed to decrease the gain, increase or decrease the depth, or control any other operating parameter.

[0046] In the case of freezing the image displayed by the ultrasound imaging subsystem 331, the recognizer 305 recognizes the lip movement of the specified command, and in response thereto, the command generator 306 generates a command to the ultrasound control subsystem 332 to signal the ultrasound imaging subsystem 331 to freeze the image being displayed on the display screen 340. It should be noted that freezing the ultrasound image can be part of the process of saving the ultrasound image.

[0047] In the case of saving the image displayed by the ultrasound imaging subsystem 331, the recognizer 305 recognizes the lip movement of the designated command, and in response thereto, the command generator 306 generates a command to the ultrasound control subsystem 332 to signal the ultrasound imaging subsystem 331 to save the image being displayed on the display screen 340. It should be noted that, in one embodiment, the image data displayed by the ultrasound imaging subsystem 331 comes from the ultrasound control subsystem 332. Therefore, in response to the command to the ultrasound control subsystem 332 to save the image being displayed on the display screen 340, the ultrasound control subsystem 332 stores the image data of the image being displayed on the display screen 340 in a memory (e.g., the memory 304).

[0048] In one embodiment, the command is recognized by the identifier 305, and in response thereto, the command generator 306 also accesses the history, workflow and / or examination data provided in the machine state information 311 and generates a command. For example, a user or doctor may be accustomed to freezing or saving an image frame after adjusting the depth of the image. As described above, using the identity of the clinician or user, after giving a command to change the depth of the image, an instruction or command to save or freeze the image can be automatically implemented, thereby combining the optimization increment with the historical data. The historical data is not limited to actions performed by only one specific clinician (i.e., individual), and can be actions of a group of individuals (e.g., other clinicians, physicians, individuals, etc. in a medical institution). In addition, the historical data may include previously used settings, parameters and / or configurations of the system, data learned by the system or other systems (e.g., data from a machine learning or artificial intelligence process, etc.), etc.

[0049] In the case of annotation, the recognizer 305 recognizes the lip movements as commands from the user to generate annotations to be placed on the image being displayed by the ultrasound imaging subsystem 331. In one embodiment, the lip movements recognized by the command generator 306 include not only the command to add the annotations to the image being displayed by the imaging subsystem 331, but also the actual annotations. It should be noted that a combination of two or more contactless inputs (e.g., lip reading and speech) can be used together to increase the accuracy of dictation recognition. In one embodiment, the recognizer 305 also recognizes the beginning and end of the annotation. In one embodiment, the recognizer 305 determines the beginning and end of the annotation by recognizing the start and stop words lip-moved by the user. In an alternative embodiment, the recognizer 305 recognizes one or more gestures (e.g., facial cues) made by the user to indicate the beginning and end of the annotation.

[0050] In one embodiment, the recognizer 305 also recognizes commands uttered by the user's lips to move annotations being displayed on the display by the ultrasound imaging subsystem 331. In one embodiment, the recognizer 305 recognizes lip movements indicating one or more commands to move the annotation up, down, left, or right on the image being displayed.

[0051] In one embodiment, the recognizer 305 also recognizes lip movements to generate commands to create a report. In one embodiment, the report can be a billing record that is created and includes a set of predetermined information necessary to ensure that the bill is processed and paid. In another embodiment, the report includes a medical record report that includes information from the ultrasound examination being performed. In one embodiment, the creation of the report includes dictating annotations into the report. In one embodiment, the dictation of the annotations is performed by causing an image of the report to be displayed on the display screen 340 by the ultrasound image subsystem 331 and instructing the ultrasound control subsystem to instruct the recognizer 305 to recognize lip movements, using the recognizer 305 to instruct one or more commands to move the annotations on the image being displayed up, down, left, and right to a specific location in the report. In addition, in one embodiment, the recognizer 305 determines the beginning and end of the dictated annotation by recognizing the start and stop words that the user is lip-moving.

[0052] In one embodiment, the command generator 306 generates an operation 320 (e.g., increase gain, decrease gain, increase depth, decrease depth, freeze image, save image, etc.) in response to the recognition results and information received from other parts of the ultrasound machine. In one embodiment, the feedback includes machine state information feedback 301 from the ultrasound control subsystem 332. In another embodiment, the feedback includes image information feedback 310 from the ultrasound imaging subsystem 331. The image information may correspond to an image being displayed on a display screen 340. This information can be used to determine the operation that the user wants to perform. For example, the ultrasound system may be able to determine a feature that the user may be interested in viewing based on a work list or examination type specified in the control state information 311. Based on the appearance of the feature in the image, the ultrasound system can use this information to affect the command being generated based on the results of lip reading. That is, the results of lip reading are affected by image features or other image information in the ultrasound system.

[0053] In one embodiment, the ultrasound system also has one or more user input devices (e.g., keyboard, cursor control device, etc.) for inputting data and allowing measurements to be taken from the display of the ultrasound display subsystem, a disk storage device (e.g., hard disk, floppy disk, compact disk (CD), digital video disk (DVD)) for storing acquired images, and a printer for printing images based on displayed data. These are also not described in detail. Figure 3 The diagram is shown in order to avoid obscuring the technology disclosed herein.

[0054] Figure 4 A block diagram of one embodiment of the recognizer 400 is shown. In one embodiment, the recognizer 400 includes processing logic in hardware (circuitry, dedicated logic, etc.), software (such as is run on a general purpose computer system or a dedicated machine), firmware, or a combination of the three.

[0055] Reference Figure 4 , the recognizer 400 includes a lip reading recognition component 401 that performs lip reading recognition in response to lip movements captured by a camera in an ultrasound system.

[0056] In one embodiment, the recognizer 400 includes an audio recognition component 402 that performs audio recognition on audio captured by a microphone of the ultrasound system.

[0057] Optionally, in an embodiment, the recognizer 400 includes an eye tracking component 403 that performs eye tracking. In one embodiment, the eye tracking component 403 determines that the user is looking directly at the camera, which triggers the lip reading recognition component 401 to run and perform recognition of lip movements captured by the camera of the ultrasound system.

[0058] In another embodiment, the recognizer 400 includes a gesture recognition component 404 that performs gesture recognition to trigger the operation of the lip reading recognition component 401 and / or the audio recognition component 402. In one embodiment, the gesture recognition component 404 recognizes one or more facial cues to trigger the operation of the lip reading recognition component 401. In one embodiment, these facial cues include the user closing their eyes for a predetermined period of time, the user blinking, the user nodding, or any other predetermined facial movement.

[0059] Figure 5 is a block diagram of an embodiment of an ultrasound system having a command generator including artificial intelligence or machine learning. Figure 5 , only shows Figure 3 A subset of components of an ultrasound system, such as a camera 301, a microphone 302, an ultrasound imaging subsystem 331, and an ultrasound control subsystem 332, is used to avoid confusion with a command generator 501 that uses artificial intelligence or machine learning to generate commands. In one embodiment, the artificial intelligence includes a neural network, such as, but not limited to, a deep learning neural network.

[0060] In one embodiment, the command generator 501 receives captured image data from the camera 301, optionally receives captured audio data 302, image information 310 fed back from the ultrasound imaging subsystem 331, and machine state information 311 fed back from the ultrasound control subsystem 332, and uses a neural network to affect the determination of operations or commands related to lip movements of an individual using the ultrasound machine.

[0061] In one embodiment, the command generator 501 generates operations using a deep learning neural network or other neural network. Deep learning is a machine learning technique that uses a representation learning method that allows the machine to be fed with raw data and determine the representations required for data classification. Deep learning determines the structure of a data set using a back-propagation algorithm, which is used to change the internal parameters of the deep learning machine (e.g., node weights). The deep learning machine can utilize various multi-layer architectures and algorithms to process raw data to identify favorable features without the need for external inputs that specify the features of interest.

[0062] Deep learning in the context of a neural network consists of many interconnected nodes called neurons. When activated, input neurons activate other neurons based on connections to other neurons controlled by machine parameters. Learning refines these machine parameters and the connections between neurons in the network so that the neural network behaves in the desired manner. In one embodiment, the deep learning network uses a convolutional neural network (CNN), which processes data using convolutional filters to locate and identify learned, observable features in the data. Each filter or layer of the CNN architecture transforms the input data to increase selectivity and invariance of the data. This abstraction of the data allows the network to focus on the features in the data it is trying to classify and ignore irrelevant background information.

[0063] In one embodiment, the deep learning network uses a convolutional neural network for image analysis as part of lip reading recognition and / or medical image analysis to identify features of ultrasound images. In one embodiment, the CNN architecture is used for facial recognition in natural images to identify the user of the ultrasound machine, thereby performing eye tracking and determining whether the user is looking at the ultrasound machine as described above. In one embodiment, the CNN architecture is also used for audio (e.g., speech) and video (e.g., lip reading) processing.

[0064] In one embodiment, the deep learning neural network includes multiple layers connected by neural connections. Data flows forward from the input layer to the output layer through the input, and then flows to the output of the deep learning neural network. The input layer including multiple nodes is followed by one or more hidden layers including nodes. After the (multiple) hidden layers is the output layer, which includes at least one node with output. In one embodiment, each input corresponds to a node of the input layer, and each node of the input layer is connected to each node of the hidden layer. Each node of a hidden layer is connected to each node of the next hidden layer, and each node of the last hidden layer is connected to the output layer. The output layer has an output to provide an output.

[0065] The CNN architecture of the deep learning neural network provides a structure that retains temporary information so that the feature information recognized by the CNN architecture is processed into an information sequence. For example, lip reading and audio information may include information sequences with different lengths, and previously recognized information is used as context information for current information to enable recognition of user commands. In one embodiment, the structure includes one or more of a recurrent neural network (RNN), a long short-term memory (LSTM), and a gated recurrent unit (GRU).

[0066] In one embodiment, the node is provided with feature data obtained from lip movements captured by the ultrasound system's camera(s).

[0067] In a neural network, certain example connections may be given increased weights, while other example connections may be given less weights. Input nodes are activated by receiving input data via the inputs. The nodes of the hidden layer are activated by the forward flow of data through the network via the connections. The nodes of the output layer are activated after the data processed in the hidden layer is sent through the connections. When the output nodes of the output layer are activated, the output nodes output appropriate values ​​based on the processing done in the hidden layer of the neural network.

[0068] In one embodiment, the neural network includes an image analysis convolutional neural network (CNN) and an RNN, LSTM, GRU or other such memory-based structure for lip reading and / or determining one or more features of an ultrasound image. The convolutional neural network receives an input image and generates image features in the form of a vector. The LSTM, RNN, GRU or other such structured network receives the image feature vector from the CNN and generates a state and output vector that is decoded into characters. The characters are used to determine whether the lip movements in the image represent a command.

[0069] Therefore, using artificial intelligence in conjunction with a camera on an ultrasound machine allows the system to read the user's lips to determine if the user is giving commands to the ultrasound machine.

[0070] In one embodiment, the command generator 501 uses a deep learning neural network to generate one or more optional commands and causes the ultrasound image subsystem 331 to display the one or more optional commands. The optional commands have associated confidence indication information that specifies the confidence level that each of the one or more commands is the desired command. In one embodiment, the confidence indication information includes a confidence level. Such confidence levels are generated using a neural network in a manner known in the art. The confidence levels are affected by the dynamic weighting of the different touch inputs used to generate the command.

[0071] In one embodiment, artificial intelligence not only determines whether the user is issuing a command, but is also used to predict the commands that the user may give to the system in the future. Thus, using the machine state and the image information 310 fed back from the ultrasound imaging subsystem, the neural network of the command generator 501 determines the operation, or influences the selection of the generated command.

[0072] For example, if a nerve in an ultrasound image is not centered in the display screen, and the clinician is inserting a needle close to the nerve to inject anesthesia, the neural network identifies the needle in the image and the exam type specified in the machine state information 311 fed back from the ultrasound control system. Upon identifying the nerve, the neural network determines a command to change the depth and center the nerve on the display screen, or biases the command determination toward a command to change the depth and center the nerve on the display screen, thereby providing ease of use in the anesthesia area. That is, lip reading command recognition associates what is displayed in the ultrasound image with what is recognized by lip reading, so that the command generation process is biased toward commands to change the depth. In this way, knowledge of the workflow is used to improve the accuracy of lip reading.

[0073] Figure 6 is a flow chart of one embodiment of a process for controlling an ultrasound machine using a neural network. In one embodiment, the process is performed by processing logic that includes a neural network implemented by the processing logic, which may include hardware (circuitry, dedicated logic, etc.), software (e.g., software running on a general purpose computer system or a dedicated machine), firmware, or a combination of the three.

[0074] Reference Figure 6 The process begins by performing lip reading, including capturing lip movements with one or more cameras and performing recognition on the lip movements (processing block 601).

[0075] The processing logic also identifies one or more image characteristics in the image being displayed by the ultrasound imaging subsystem that are not optimized (processing block 602) and associates operating parameters that will change the image to help improve and potentially optimize the image feature(s) (processing block 603). This may occur prior to or concurrently with lip reading.

[0076] In response to performing the lip reading and after associating the operating parameters that will change the image, processing logic uses a neural network (e.g., a deep learning neural network) to determine an operation for controlling the ultrasound machine (processing block 604). The neural network determines the operation based on the results of the lip reading recognition, machine state information of the ultrasound machine, and based on the association with the operating parameters for changing the image. In one embodiment, the change to the image is an optimization, improvement, restoration, or other modification of the image. That is, in one embodiment, as part of the process of determining the operation, the neural network is associated with the operating parameters for the optimization, improvement, restoration, or other modification of the image.

[0077] In one embodiment, after determining one or more operations based on the results of lip reading recognition, machine state information of the ultrasound machine, and association with operating parameters for changing the image, processing logic displays one or more user-selectable operations that can be performed by the ultrasound machine (processing block 605). It should be noted that this is an optional step and is not required. In an alternative embodiment, a single operation is determined and the ultrasound machine generates a command for execution without selection or confirmation.

[0078] Displaying the operation that has been determined based on the input enables the user to select and / or confirm the operation(s) to be performed by the ultrasound machine, which is advantageous in situations where the command generator cannot unambiguously determine the operation the user wants to be performed. This is confirmation mode.

[0079] In one embodiment, the optional operations are presented on a display of the ultrasound machine under control of the ultrasound imaging subsystem. In one embodiment, the optional operations are displayed in a list based on their likelihood of matching a user desired operation determined by a command generator of the ultrasound machine. In one embodiment, the list of optional operations includes a confidence factor generated by the command generator for each operation, or other information providing the user with an indication of the confidence level associated with the determination of the ultrasound machine operation.

[0080] Next, processing logic uses the operation to control the ultrasound machine (processing block 606). In one embodiment, control is responsive to selection and / or confirmation of the operation, if necessary. In one embodiment, control is performed by a control subsystem of the ultrasound machine.

[0081] It should be noted that except Figure 6 Other operations beyond those specified in may be determined and used to control the ultrasound machine, such as those discussed above.

[0082] Various example embodiments are described herein.

[0083] Example 1 is a method for controlling the operation of an ultrasound machine, the method comprising: obtaining one or more non-contact inputs; determining one or more operations for controlling the ultrasound machine based on the one or more non-contact inputs and a machine state of the ultrasound machine; and controlling the ultrasound machine using at least one of the one or more operations.

[0084] Example 2 is a method of Example 1 that may optionally include the following features: including performing lip reading, wherein performing lip reading includes capturing lip movements of an individual using at least one camera, and wherein determining one or more operations of controlling the ultrasound machine is based on the results of performing lip reading.

[0085] Example 3 is the method of Example 1 which may optionally include the following feature: one or more operations are further determined based on the ultrasound data.

[0086] Example 4 is the method of Example 1 which may optionally include the following features: the ultrasound data includes one or more of ultrasound image data, a type of examination being performed using the ultrasound machine, and a list of operations being performed by the ultrasound machine.

[0087] Example 5 is the method of Example 1 which may optionally include the following feature: determining one or more operations further includes predicting the operations that the individual is likely to perform based on historical data.

[0088] Example 6 is a method of Example 1 that may optionally include the following features: determining one or more operations further based on the captured individual audio information, and including dynamically adjusting weights associated with the results of lip reading recognition and audio recognition based on the environment in which the ultrasound machine is located to determine one or more operations to control the ultrasound machine.

[0089] Example 7 is a method of Example 1 that may optionally include the following features: triggering lip reading in response to determining that the individual is looking directly at one of the one or more cameras, or in response to determining that the individual closes at least one eye for a predetermined period of time, blinks, nods, or performs another facial cue or other gesture.

[0090] Example 8 is a method of Example 1 that may optionally include the following features: identifying one or more image features, and wherein determining one or more operations is further based on an association of operating parameters with the one or more image features, and further wherein using at least one of the one or more operations to control the ultrasound machine includes changing the operating parameters to change the image.

[0091] Example 9 is a method of Example 1 that may optionally include the following features: receiving one or more of the following items at a neural network: ultrasound data feedback from an ultrasound imaging subsystem, machine state feedback from an ultrasound control subsystem, at least one of one or more non-contact inputs, and wherein determination of one or more operations to control the ultrasound machine is performed by the neural network based on the one or more non-contact inputs and the machine state of the ultrasound machine.

[0092] Example 10 is a method of Example 1 that may optionally include the following features: determining one or more operations for controlling an ultrasound machine includes generating and displaying one or more optional commands; capturing information from an individual; and interpreting the captured information as confirmation of selection of at least one of the one or more optional commands before using at least one operation to control the ultrasound machine.

[0093] Example 11 is a method of Example 1 that may optionally include the following features: one or more operations include at least one selected from the group consisting of adjusting gain, adjusting depth, freezing an image being displayed by an ultrasound machine, saving an image being displayed by an ultrasound machine, adding annotations at a user-specified location on an image being displayed by an ultrasound machine, and creating a report using one or more images displayed by the ultrasound machine.

[0094] Example 12 is the method of Example 1 which may optionally include the following feature: determining one or more operations to control the ultrasound machine includes: placing an annotation on an image being displayed by the ultrasound machine based on at least one of the one or more non-contact inputs.

[0095] Example 13 is a method that may optionally include Example 12 as characterized: wherein determining one or more operations to control the ultrasound machine further includes identifying a start of an annotation command, an end of an annotation command, and one or more annotation move commands based on at least one non-contact input.

[0096] Example 14 is the method of Example 12 which may optionally include identifying one or more annotation movement commands from the eye tracking information.

[0097] Example 15 is a method of Example 1 that may optionally include the following features: based on a user identification operation, identifying an individual from a group of individuals located near the ultrasound machine; and providing control of the ultrasound machine to the identified individual by using a contactless input.

[0098] Example 16 is a device comprising: a display screen; an ultrasound imaging subsystem coupled to the display to generate an ultrasound image on the display screen; an ultrasound control subsystem coupled to control the imaging subsystem; one or more cameras to capture images of an individual's lip movements; a microphone to capture audio; a recognizer coupled to the one or more cameras to perform a lip recognition program on the captured images from the one or more cameras to perform lip reading; and a command generator coupled to the recognizer and control subsystem to determine one or more operations based on one or more non-contact inputs from the recognizer and the microphone and a machine state of the ultrasound machine received from the ultrasound control subsystem, and to send at least one of the one or more operations to control the ultrasound control subsystem.

[0099] Example 17 is the device of Example 16 which may optionally include the following features: the command generator includes a neural network.

[0100] Example 18 is the apparatus of Example 16 which may optionally include the following features: the command generator is operable to determine one or more operations further based on the ultrasound data.

[0101] Example 19 is the device of Example 18 which may optionally include the following features: the ultrasound data includes one or more of ultrasound image data, a type of examination being performed using the ultrasound machine, and a list of operations being performed by the ultrasound machine.

[0102] Example 20 is the device of Example 16 which may optionally include the following features: the command generator is operable to determine the one or more operations by predicting the operations that the individual is likely to perform based on historical data.

[0103] Example 21 is a device of Example 16 that may optionally include the following features: the command generator is operable to further determine one or more operations based on the captured individual audio information, and dynamically adjust the weights associated with the results of lip reading recognition and audio recognition according to the environment in which the ultrasound machine is located to determine one or more operations to control the ultrasound machine.

[0104] Example 22 is a device of Example 16 that may optionally include the following features: the identifier is operable to trigger a contactless input control for use by the individual in response to determining that the individual is looking at one of one or more cameras or in response to a facial prompt, wherein the facial prompt includes one of the group consisting of: closing at least one eye for a predetermined period of time, blinking, nodding.

[0105] Example 23 is a device of Example 16 that may optionally include the following features: the command generator includes a neural network that is operable to receive ultrasound data feedback from the ultrasound imaging subsystem, machine state feedback from the ultrasound control subsystem, and at least one of one or more non-contact inputs from an individual of the identifier, and in response thereto, the command generator is operable to determine one or more operations to control the ultrasound control subsystem.

[0106] Example 24 is a device of Example 16 that may optionally include the following features: one or more operations include at least one selected from the group consisting of adjusting gain, adjusting depth, freezing an image being displayed by an ultrasound machine, saving an image being displayed by an ultrasound imaging subsystem, adding annotations at a user-specified location on an image being displayed by the ultrasound imaging subsystem, and creating a report using one or more images displayed by the ultrasound imaging subsystem.

[0107] Example 25 is a device of Example 16 that may optionally include the following features: one or more operations include placing an annotation on an image being displayed by an ultrasound imaging subsystem based on one or more non-contact inputs, including identifying the beginning of an annotation command, the end of an annotation command, and one or more annotation move commands based on information captured from the individual.

[0108] Example 26 is an article of manufacture having one or more computer-readable storage media storing instructions, which, when executed by a system, cause the system to perform a method for controlling the operation of an ultrasound machine, wherein the method includes: obtaining one or more non-contact inputs; determining one or more operations for controlling the ultrasound machine based on the one or more non-contact inputs and a machine state of the ultrasound machine; and using at least one of the one or more operations to control the ultrasound machine.

[0109] Some portions of the above detailed description are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here generally considered to be a self-consistent sequence of steps for achieving a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, although not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It proves convenient at times, primarily for common usage reasons, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0110] It should be remembered, however, that all of these and similar terms are associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise clearly indicated from the following discussion, it should be understood that throughout the specification, discussions using terms such as "processing," "computing," "determining," or "displaying" refer to the actions and processes of a computer system or similar electronic computing device that manipulates data represented as physical (electronic) quantities in the computer system's registers and memories and converts them into data similar to other data represented as physical quantities in the computer system's memories or registers or other such information storage, transmission, or display devices.

[0111] The present invention also relates to an apparatus for performing the operations described herein. The apparatus may be specially constructed for the desired purpose, or it may include a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer-readable storage medium, such as, but not limited to, any type of disk including a floppy disk, an optical disk, a CD-ROM, and a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic card or an optical card, or any type of medium suitable for storing electronic instructions and each connected to a computer system bus.

[0112] The algorithms and displays provided herein are not inherently related to any particular computer or other device. Various general purpose systems can be used together with programs according to the teachings herein, or it may be convenient to construct more specialized equipment to perform the required method steps. The required structures of various these systems will be seen from the following description. In addition, the present invention is not described with reference to any specific programming language. It should be understood that various programming languages ​​can be used to implement the teachings of the present invention as described herein.

[0113] Machine-readable media include any mechanism (e.g., a computer) for storing or transmitting information in a form readable by a machine. For example, machine-readable media include read-only memory ("ROM"); random access memory ("RAM"); magnetic disk storage media; optical storage media; flash memory devices; etc.

[0114] Although many changes and modifications of the present invention will undoubtedly become apparent to those of ordinary skill in the art after reading the foregoing description, it should be understood that any particular embodiment shown and described by way of example is in no way intended to be considered limiting. Therefore, reference to details of various embodiments is not intended to limit the scope of the claims, which themselves recite only those features deemed essential to the present invention.

Claims

1. A method for controlling the operation of an ultrasound machine, the method comprising: obtaining one or more contactless inputs, including lip reading recognition data of a user and captured audio recognition data; determining one or more operations for controlling the ultrasound machine based on the user's lip reading recognition data, the captured audio recognition data, and information about the current state of the ultrasound machine, wherein the lip reading recognition data and the captured audio recognition data are weighted to vary their respective contributions to determining the one or more operations for controlling the ultrasound machine based on a noise level of an environment in which the ultrasound machine is located, such that if the noise level is above a noise threshold, the lip reading recognition data is weighted higher than the captured audio recognition data, and if the noise level is below the noise threshold, the lip reading recognition data is weighted less than the captured audio recognition data; and The ultrasound machine is controlled using at least one of the one or more operations.

2. The method according to claim 1, further comprising: Lip reading is performed, including capturing lip movements of a user using at least one camera, and wherein determining the one or more operations of controlling the ultrasound machine is based on a result of performing the lip reading. The method of claim 1 , wherein determining the one or more operations is further based on ultrasound data. 4 . The method of claim 3 , wherein the ultrasound data comprises one or more of ultrasound image data, a type of examination being performed by the ultrasound machine, and a list of operations performed by the ultrasound machine. The method of claim 1 , wherein determining the one or more operations comprises predicting operations that the user is likely to perform based on historical data.

6. The method of claim 1 , further comprising dynamically adjusting weights associated with the lip reading recognition data and the captured audio recognition data based on a noise level of an environment in which the ultrasound machine is located to determine the one or more operations to control the ultrasound machine.

7. The method of claim 2, wherein lip reading is triggered in response to determining that the user is looking directly at one of the one or more cameras, or in response to determining that the user has closed at least one eye for a predetermined period of time, blinked, nodded, or performed another facial cue or other gesture.

8. The method of claim 1 , further comprising identifying one or more image features in an image displayed by the ultrasound machine and associating operating parameters of the ultrasound machine with the one or more image features, wherein controlling the ultrasound machine using at least one of the one or more operations comprises changing the operating parameters to change the image.

9. The method of claim 1 , wherein the ultrasound machine comprises a neural network that receives the lip reading recognition data and the captured audio recognition data, one or more image information from an ultrasound imaging subsystem, and information about a current state of an ultrasound control subsystem, and wherein the determination of the one or more operations to control the ultrasound machine is performed by the neural network based on the user's lip reading recognition data, the captured audio recognition data, and the information about the current state of the ultrasound machine.

10. The method of claim 1, wherein determining the one or more operations to control the ultrasound machine comprises: Generate and display one or more optional commands; as well as Prior to controlling the ultrasound machine using at least one of the one or more operations, receiving a user selection of at least one of the one or more selectable commands.

11. The method according to claim 1, wherein the one or more operations include at least one selected from the group consisting of: adjusting gain, adjusting depth, freezing the image being displayed by the ultrasound machine, saving the image being displayed by the ultrasound machine, adding annotations at a user-specified location on the image being displayed by the ultrasound machine, and creating a report using one or more images displayed by the ultrasound machine.

12. The method of claim 1, wherein determining the one or more operations to control the ultrasound machine comprises: Based on at least one of the one or more non-contact inputs, an annotation is placed on an image being displayed by the ultrasound machine.

13. The method of claim 12, wherein determining the one or more operations to control the ultrasound machine comprises: Based on at least one contactless input, determining a start of the annotation, an end of the annotation, and determining one or more commands to move the annotation.

14. The method of claim 12, wherein one or more commands to move the annotation are identified from eye tracking information.

15. The method according to claim 1, further comprising: Based on a user identification operation, identifying the user located near the ultrasound machine from a group of individuals; and Control of the ultrasound machine is provided to the identified user using the contactless input.

16. A device comprising: Display screen; an ultrasound imaging subsystem coupled to the display screen to generate an ultrasound image on the display screen; an ultrasound control subsystem coupled to control the ultrasound imaging subsystem; one or more cameras to capture images of the user's lip movements; Microphone, to capture audio; a recognizer coupled to the one or more cameras to perform a lip recognition routine on captured images from the one or more cameras to perform lip reading and to perform an audio recognition routine on captured audio of a user; and A command generator coupled to the recognizer and the ultrasound control subsystem to determine one or more operations based on one or more contactless inputs including lip reading recognition data and captured audio recognition data from a user of the recognizer and information about a current state of the ultrasound machine received from the ultrasound control subsystem, and to send at least one of the one or more operations to control the ultrasound control subsystem, wherein the lip reading recognition data and the captured audio recognition data are weighted to change their respective contributions to determining the one or more operations for controlling the ultrasound machine based on a noise level of an environment in which the ultrasound machine is located, such that if the noise level is above a noise threshold, the lip reading recognition data is weighted higher than the captured audio recognition data, and if the noise level is below the noise threshold, the lip reading recognition data is weighted less than the captured audio recognition data.

17. The apparatus of claim 16, wherein the command generator comprises a neural network.

18. The apparatus of claim 16, wherein the command generator is operable to determine the one or more operations based further on ultrasound data.

19. The apparatus of claim 18, wherein the ultrasound data comprises one or more of ultrasound image data, a type of examination performed by the ultrasound machine, and a list of operations being performed by the ultrasound machine.

20. The apparatus of claim 16, wherein the command generator is operable to determine the one or more operations by predicting operations that the user is likely to perform based on historical data.

21. The apparatus of claim 16, wherein the command generator is operable to dynamically adjust weights associated with the lip reading recognition data and the captured audio recognition data based on a noise level of an environment in which the ultrasound machine is located to determine the one or more operations to control the ultrasound machine.

22. The apparatus of claim 16, wherein the identifier is operable to trigger a touchless input control used by the user in response to determining that the user is looking at one of the one or more cameras or in response to a facial prompt, wherein the facial prompt comprises one of the group consisting of: closing at least one eye for a predetermined period of time, blinking, nodding.

23. The apparatus of claim 16, wherein the command generator comprises a neural network operable to receive the lip reading recognition data and the captured audio recognition data, one or more image information from the ultrasound imaging subsystem, and information about a current state of the ultrasound control subsystem, and in response thereto, the command generator is operable to determine one or more operations to control the ultrasound control subsystem.

24. The device of claim 16, wherein the one or more operations include at least one selected from the group consisting of adjusting gain, adjusting depth, freezing an image being displayed by the ultrasound machine, saving an image being displayed by the ultrasound imaging subsystem, adding annotations at a user-specified location on the image being displayed by the ultrasound imaging subsystem, and creating a report using one or more images displayed by the ultrasound imaging subsystem.

25. The apparatus of claim 16, wherein the one or more operations comprise: An annotation is placed on an image being displayed by the ultrasound imaging subsystem based on the one or more non-contact inputs, and wherein determining the one or more operations to control the ultrasound machine includes determining the beginning of the annotation, the end of the annotation, and determining one or more commands to move the annotation based on information captured from the user.

26. An article of manufacture having one or more computer-readable storage media storing instructions that, when executed by a system, cause the system to perform a method for controlling operation of an ultrasound machine, the method comprising: obtaining one or more contactless inputs, including lip reading recognition data of a user and captured audio recognition data; determining one or more operations to control the ultrasound machine based on the user's lip reading recognition data, the captured audio recognition data, and information about the current state of the ultrasound machine, wherein the lip reading recognition data and the captured audio recognition data are weighted to vary their respective contributions to determining the one or more operations to control the ultrasound machine based on a noise level of an environment in which the ultrasound machine is located, such that if the noise level is above a noise threshold, the lip reading recognition data is weighted higher than the captured audio recognition data, and if the noise level is below the noise threshold, the lip reading recognition data is weighted less than the captured audio recognition data; and The ultrasound machine is controlled using at least one of the one or more operations.

Images