Optical detection of object communication requests

Abstract

The present invention provides a magnetic resonance imaging system (100, 300, 500). The magnetic resonance imaging system includes a video system (122, 122', 122") configured to provide a video image (146) of a facial region (119) of a subject (118) during acquisition of magnetic resonance imaging data (144). The magnetic resonance imaging system also includes a memory (134) for storing machine executable instructions (140) and pulse sequence commands (142). The execution of the machine executable instructions instructs a processor (130) to: control (200) the magnetic resonance imaging system to acquire magnetic resonance imaging data using the pulse sequence commands; receive (202) a video image from the video system during execution of the pulse sequence commands; determine (204) a voice activity state (150) in real time for the video image by inputting the video image into a video-based voice activity detection module, wherein the voice activity state indicates a speaking state of the subject or a non-speaking state of the subject; and, if the voice activity state indicates a speaking state, provide (206) a voice activity signal (152).

Description

Technical Field

[0001] The present invention relates to magnetic resonance imaging and, in particular, to communication between an operator of a magnetic resonance imaging system and a subject being imaged in the magnetic resonance imaging system. Background Art

[0002] Magnetic resonance imaging (MRI) scanners use a large static magnetic field to align the nuclear spins of atoms as part of the process used to produce images of the inside of a subject's body. This large static magnetic field is called the B0 magnetic field or main magnetic field. Various values ​​or properties of a subject can be measured spatially using MRI. However, the acquisition of MRI data is not instantaneous. The subject may need to remain as still as possible for several minutes. To ensure the subject's safety, they are provided with a squeeze bulb or pressure-operated balloon switch that they can squeeze if the subject needs to contact the operator. For example, suppose the subject feels pain, discomfort, or panic due to noise and / or being in a confined space.

[0003] U.S. Patent Application Publication US 2011 / 0230755 discloses an optically based rigid body 6-DOF motion tracking system that is optimized for predictive (real-time) motion correction in magnetic resonance imaging (MRI) applications using a single camera fixed to a patient with an onboard image processor, an IR illuminator, and an optical fiducial target. An angle extraction algorithm operated by the onboard image processor utilizes a successive approximation method to solve a 3-point pose problem of angles close to an origin to achieve convergence to a sub-microradian level. Motion alerts are enabled through a monitor and GUI application that communicates with the motion tracking system. Motion correction is enabled for MR scan images captured while operating the motion tracking system, wherein the MRI controller communicates with the motion tracking system.

[0004] US patent application US2005 / 0283068 discloses a magnetic resonance imaging system with a predictive noise cancellation system. The predictive noise cancellation system uses its repetitive nature to cancel the acoustic noise of the gradient system. Removing MRI acoustic noise from the patient microphone enables the technician to hear clear speech from the patient in the presence of very strong noise presented by the MRI magnet. Summary of the invention

[0005] The invention provides a magnetic resonance imaging system, a computer program product, and a method in the independent claims. Embodiments are given in the dependent claims.

[0006] A difficulty with using a squeeze bulb to alert an operator of a magnetic resonance imaging system is that the subject may inadvertently drop the bulb and then panic or begin moving in an attempt to regain the bulb. Holding the squeeze bulb may also be difficult for children or the elderly. Embodiments of the present invention provide an alternative to using a squeeze bulb by providing a video system that images a facial region of a subject during acquisition of magnetic resonance imaging data. The video system provides a video image that is input into a video-based voice activity detection module that detects when a subject is speaking. When it is determined using the video-based voice activity detection module that the subject is speaking, a voice activity signal is provided.

[0007] In one aspect, the present invention provides a magnetic resonance imaging system configured to acquire magnetic resonance imaging data from an object at least partially within an imaging region. The magnetic resonance imaging system includes a video system configured to provide a video image of a facial region of the object. In some cases, this may be during the acquisition of the magnetic resonance imaging data, or even before the acquisition of the magnetic resonance imaging data. The magnetic resonance imaging system also includes a memory for storing machine executable instructions and pulse sequence commands. The pulse sequence commands are configured to control the magnetic resonance imaging system to acquire magnetic resonance imaging data according to a magnetic resonance imaging protocol. The pulse sequence commands may, for example, be commands or data that may be converted into these commands that enable a processor to control various components to acquire magnetic resonance imaging data.

[0008] The magnetic resonance imaging system also includes a processor for controlling the magnetic resonance imaging system. Execution of the machine executable instructions causes the processor to control the magnetic resonance imaging system to acquire magnetic resonance imaging data using pulse sequence commands. Execution of the machine executable instructions also causes the processor to receive video images from the video system during acquisition of the magnetic resonance imaging data. Execution of the machine executable instructions also causes the processor to determine a voice activity state for the video image in real time by inputting the video image into a video-based voice activity detection module. The voice activity state indicates a speaking state of the subject or a non-speaking state of the subject. If the voice activity state indicates a speaking state, execution of the machine executable instructions also causes the processor to provide a voice activity signal.

[0009] This embodiment may be beneficial because it may provide a way of identifying when a subject is experiencing discomfort or having problems during the execution of machine executable instructions to acquire magnetic resonance imaging data. Typically, currently the subject is provided with a light bulb which they are able to squeeze in their hand to signal the operator of the magnetic resonance imaging system. This may be a problem for the subject who may, for example, lose their grip on the bulb and be unable to regain it. Additionally, the subject may be stressed about maintaining their grip on the squeeze bulb. If the subject in this embodiment is having difficulty or is experiencing pain, the subject may simply begin to speak which will provide a voice activity signal which may, for example, be used to alert the operator.

[0010] In some examples, there may be a threshold of how many frames are marked before voice activity is activated.

[0011] In typical video-based voice activity detectors, even if they are properly trained, they may use short clips (e.g., half a second in duration) or a series of frames (e.g., 25 frames in the case of a frame rate of 50 frames) to determine whether a subject is speaking. In some examples, criteria for a specific duration or number of frames may be used to indicate speaking. In some cases, a specific number of frames or a duration before a specific frame is used to decide whether a frame should be marked as a speaking state or a non-speaking state. References to individual images captured by a video system may, for example, be referred to as video images, frames for a single image, or also as frames for multiple video images.

[0012] The facial region may, for example, include the area where the mouth is visible. In some examples, this may include the teeth and lips of the subject.

[0013] In another embodiment, the magnetic resonance imaging system includes a communication system. The communication system can be configured to be activated when a sound activity signal is provided. In some examples, the communication system can provide audio and / or video communication between the subject and the operator.

[0014] In one embodiment, the communication system is a walkie-talkie.

[0015] In another embodiment, the magnetic resonance imaging system further comprises an object microphone, which is configured to obtain audio data from the object. Including the object microphone can be beneficial because it can provide an operator with a way to speak and communicate directly with the object. In other examples, the use of the object microphone can be beneficial because it can provide a way to train a video-based sound activity detection module for a specific object. In other examples, this can be beneficial because it can be used to record what the object said during the acquisition of the magnetic resonance imaging data. For example, it can be useful to play back the audio from the object after the sound activity signal has been provided. This can, for example, provide the operator with an opportunity to ignore the object's voice or even determine the need to shut down the system immediately before speaking or communicating with the object.

[0016] In another embodiment, execution of the machine executable instructions further causes the processor to record an initial video image of the subject using a video system. The initial video image includes frames. Execution of the machine executable instructions further causes the processor to record initial audio of the subject using a subject microphone while recording the initial video image. Execution of the machine executable instructions further causes the processor to mark the frames as speaking or not speaking by inputting the initial audio into an audio-based voice activity detector module. Execution of the machine executable instructions further causes the processor to train a video-based voice activity detection module using the frames and the markings of each frame. In this embodiment, the video-based voice activity detection module can be trained by collecting audio and video from the subject. For example, the video-based voice activity detection module can be pre-trained, but additional training can also be used to enhance its ability to detect the speech of a specific subject.

[0017] In another embodiment, the training of the video-based voice activity detection module is performed using deep learning.For example, the video-based voice activity detection module may be a neural network or a so-called convolutional neural network.

[0018] In another embodiment, the audio-based sound activity detector module is configured to exclude gradient coil noise during the marking of frames as speaking or not speaking. For example, the sound activity detector module can use a filter, or can be trained to ignore the clicking and thumping that the gradient coil may use. For example, when performing a magnetic resonance imaging protocol, in many preparation stages for the first scan, it is either relatively quiet or a low noise consisting of short clicks that are usually reproducible for various protocols is generated. The sounds determined by the resonant frequency are some well-defined clicks that last for about five seconds. Such a time period can be used, for example, to collect initial video images and initial audio of the object to generate training data. Therefore, it can be proposed to pre-train and configure the audio-based sound activity detector algorithm to ignore these sounds that may be emitted by the gradient coil system.

[0019] In another embodiment, the magnetic resonance imaging system further comprises a prompter configured to present questions to the subject. Execution of the machine executable instructions further causes the processor to present a list of questions to the subject using the prompter during recording of the initial video image and the initial audio. This may be beneficial, for example, because it may induce speech from the subject, which speech can be recorded as the initial video image and the initial audio. This may also have the advantage that various questions that an operator might ask the subject will be automatically asked, and useful information or metadata about the subject will be obtained in addition to providing training for the video-based voice activity detection module.

[0020] The prompter can be, for example, a video display prompter that generates questions that the subject can read. In other cases, the prompter can be an audio system that can play pre-recorded or synthetic sounds that pose questions to the subject.

[0021] In some embodiments, the video system may include a camera for imaging the subject while the subject is outside the magnetic resonance imaging system. For example, if there is a subject support capable of moving the subject into the magnet, the subject may be able to have his or her face imaged, for example, before the subject enters the magnetic resonance imaging system. This may enable acquisition of initial video images and / or initial audio before the subject is even placed into the magnetic resonance imaging system.

[0022] In another embodiment, the magnetic resonance imaging system further comprises a subject support configured to support at least a portion of the subject in the imaging region. The subject microphone is incorporated into the subject support. This may be beneficial as it may provide a convenient way to record audio data from the subject.

[0023] In another embodiment, the execution of the machine executable instructions further causes the processor to continuously record audio data using the subject microphone during the acquisition of the magnetic resonance imaging data. The audio data is synchronized with the video image. The execution of the machine executable instructions further causes the processor to mark the audio data with a sound activity state, which is determined using a video-based sound activity detection module. When a signal from a user interface is received, the execution of the machine executable instructions further causes the processor to play back the audio data marked with a speaking state. For example, after the subject is detected to be speaking using the video system, the audio recorded when the subject is detected to be speaking can be controllably played back to the operator of the magnetic resonance imaging system. This can help the operator make a decision whether to stop the acquisition of magnetic resonance data or perform some other action.

[0024] In another embodiment, execution of the machine executable instructions further causes the processor to receive a spoken word list by inputting the audio data into a speech recognition module. For example, the recorded audio data may be converted to speech and then tokenized or tabulated. Execution of the machine executable instructions further causes the processor to perform any of the following operations: if at least a portion of the spoken word list matches a predetermined word list, then raise a predetermined alarm in a user interface of the magnetic resonance imaging system. Alternatively or additionally, if at least a portion of the spoken word list matches the predetermined word list, then the processor may stop the acquisition of magnetic resonance imaging data. This embodiment may be beneficial because the magnetic resonance imaging system may be programmed to take appropriate action if a specific word is indicated, for example if a subject indicates pain or speaks a word like burning during the acquisition of magnetic resonance imaging data that may indicate a serious problem. In this case, the magnetic resonance imaging system may be able to stop itself faster than the operator.

[0025] In another embodiment, execution of the machine executable instructions further causes the processor to provide an indicator as a signal on a user interface. The user interface is configured to open a communication channel between an operator and the subject.

[0026] In another embodiment, execution of the machine executable instructions may further cause the processor to generate an audio tone if a voice activity signal is provided.

[0027] In another embodiment, the video-based voice activity detection module includes a convolutional neural network. This can be beneficial because convolutional neural networks are good at recognizing various patterns that may be present in audio and / or video.

[0028] In another embodiment, the magnetic resonance imaging system comprises a magnet having a bore. The camera system comprises a plurality of cameras mounted within the bore. This may have the advantage that the position where the subject is placed is not important. If there are a plurality of different cameras, the operator or the system itself can select the camera that provides the best view of the facial area of ​​the subject.

[0029] It is possible to provide very small and compact cameras. It may be practical to place a large number of cameras within the bore of the magnet.

[0030] In another embodiment, the magnetic resonance imaging system includes a head coil. The camera system includes a coil-mounted camera attached to the head coil. This embodiment can be beneficial because the head coil can be designed to fit the subject in a certain way. Thus, the camera can be provided in an optimal manner to image the facial area of ​​the subject. Likewise, the data bus for the head coil can also be shared with the coil-mounted camera.

[0031] In another embodiment, the camera system includes a flange-mounted camera configured to image a reflected image of a facial region of the subject. For example, there may be a mirror mounted on the magnetic resonance imaging system, or even a head coil or other device placed on the subject's head that can be used to direct an image of the subject's facial region into the flange-mounted camera.

[0032] In another aspect, the present invention provides a method of operating a magnetic resonance imaging system, the magnetic resonance imaging system being configured to acquire magnetic resonance imaging data from a subject at least partially within an imaging region. The magnetic resonance imaging system includes a video system, the video system being configured to provide a video image of a facial region of the subject during acquisition of the magnetic resonance imaging data. The method includes controlling the magnetic resonance imaging system to acquire the magnetic resonance imaging data using pulse sequence commands. The pulse sequence commands are configured to control the magnetic resonance imaging system to acquire the magnetic resonance imaging data. The method also includes receiving a video image from the video system. This is done during acquisition of the magnetic resonance imaging data.

[0033] The method also includes determining a voice activity state in real time for the video image by inputting the video image into a video-based voice activity detection module. The voice activity state indicates a speaking state of the subject or a non-speaking state of the subject. The method also includes providing a voice activity signal if the voice activity state indicates a speaking state. The advantages of this embodiment have been previously discussed.

[0034] In another aspect, the present invention provides a computer program product comprising machine executable instructions for execution by a processor controlling a magnetic resonance imaging system, the magnetic resonance imaging system being configured to acquire magnetic resonance imaging data from an object at least partially within an imaging region. Execution of the machine executable instructions further causes the processor to receive a video image from a video system during acquisition of the magnetic resonance imaging data. Execution of the machine executable instructions further causes the processor to determine a voice activity state in real time using the video image by inputting the video image into a video-based voice activity detection module.

[0035] The voice activity state indicates a speaking state of the subject or a non-speaking phase of the subject. If the voice activity state indicates a speaking state, execution of the machine executable instructions further causes the processor to provide a voice activity signal. This embodiment can be used, for example, as a plug-in or additional software for an existing magnetic resonance imaging system to which the functionality previously described above can be added. The advantages have been described previously.

[0036] In another aspect, the invention provides a computer program product comprising machine executable instructions executed by a processor controlling a magnetic resonance imaging system, the magnetic resonance imaging system being configured to acquire magnetic resonance imaging data from a subject at least partially within an imaging region. The magnetic resonance imaging system comprises a video system configured to provide a video image of a facial region of the subject during acquisition of the magnetic resonance imaging data. Execution of the machine executable instructions causes the processor to control the magnetic resonance imaging system to acquire the magnetic resonance imaging data using pulse sequence commands.

[0037] The pulse sequence commands are configured to control the magnetic resonance imaging system to acquire magnetic resonance imaging data. Execution of the machine executable instructions further causes the processor to receive a video image from the video system during acquisition of the magnetic resonance imaging data. Execution of the machine executable instructions further causes the processor to determine a voice activity state for the video image in real time by inputting the video image into a video-based voice activity detection module. The voice activity state indicates a speaking state of the subject or a non-speaking state of the subject. If the voice activity state indicates a speaking state, execution of the machine executable instructions further causes the processor to provide a voice activity signal.

[0038] As used herein, the term "real-time" indicates that the voice activity state is determined within a predetermined delay after a specific frame or video image has been acquired. The determination of the speaking or non-speaking state of a subject will typically be based on data from one or more images, video images or frames. To this end, although the voice activity state is determined in real time, the determination may be determined, for example, at discrete time intervals.

[0039] In some embodiments, the term "real time" means that the voice activity state is determined within half a second. In other examples, the term "real time" can indicate that the voice activity state is determined within one second or two seconds.

[0040] It should be understood that one or more of the aforementioned embodiments of the present invention may be combined as long as the combined embodiments are not mutually exclusive.

[0041] As will be appreciated by those skilled in the art, aspects of the present invention may be embodied as an apparatus, method, or computer program product. Thus, aspects of the present invention may take the form of a fully hardware embodiment, a fully software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects that may be generally referred to herein as a "circuit," "module," or "system." Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer-readable media having executable code embodied thereon.

[0042] Any combination of one or more computer-readable media can be utilized. Computer-readable media can be computer-readable signal media or computer-readable storage media. As used herein, "computer-readable storage media" encompasses any tangible storage media that can store instructions that can be executed by a processor of a computing device. Computer-readable storage media can be referred to as computer-readable non-temporary storage media. Computer-readable storage media can also be referred to as tangible computer-readable media. In some embodiments, computer-readable storage media can also be capable of storing data that can be accessed by a processor of a computing device. Examples of computer-readable storage media include, but are not limited to, floppy disks, magnetic hard disk drives, solid-state hard disks, flash memory, USB thumb drives, random access memories (RAM), read-only memories (ROM), optical disks, magnetic optical disks, and register files of processors. Examples of optical disks include high-density disks (CDs) and digital versatile disks (DVDs), such as CD-ROMs, CD-RWs, CD-Rs, DVD-ROMs, DVD-RWs, or DVD-R disks. The term computer-readable storage media also refers to various types of recording media that can be accessed by a computer device via a network or communication link. For example, data can be retrieved via a modem, the Internet, or a local area network. Computer executable code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0043] Computer readable signal media may include, for example, a propagated data signal having computer executable code embodied therein, in baseband or as part of a carrier wave. Such propagated signals may take any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. Computer readable signal media may be any computer readable medium that is not a computer readable storage medium and is capable of communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, device, or apparatus.

[0044] "Computer storage" or "storage" is an example of a computer-readable storage medium. Computer storage is any memory directly accessible to a processor. "Computer memory" or "memory" is another example of a computer-readable storage medium. Computer memory is any non-volatile computer-readable storage medium. In some embodiments, computer memory may also be computer storage and vice versa.

[0045] As used herein, "processor" encompasses an electronic component capable of executing a program or machine executable instructions or computer executable code. References to a computing device that includes a "processor" should be interpreted as possibly including more than one processor or processing core. The processor may, for example, be a multi-core processor. A processor may also refer to a collection of processors within a single computer system or distributed between multiple computer systems. The term computing device should also be interpreted as possibly referring to a collection or network of computing devices that each include one or more processors. Computer executable code may be executed by multiple processors that may be within the same computing device or even distributed on multiple computing devices.

[0046] The computer executable code may include machine executable instructions or programs that cause a processor to perform an aspect of the present invention. The computer executable code for performing the operations of various aspects of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages ​​that are compiled into machine executable instructions. In some cases, the computer executable code may be in the form of a high-level language, or in a pre-compiled form, and may be used in conjunction with a compiler that generates machine executable instructions on the fly.

[0047] The computer executable code may be executed entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or a connection may be established with an external computer (e.g., through the Internet using an Internet service provider).

[0048] Various aspects of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box or part of a box of a flowchart, schematic diagram and / or block diagram can be implemented by computer program instructions in the form of computer executable code when applicable. It should also be understood that when not mutually exclusive, the combination of boxes in different flowcharts, schematic diagrams and / or block diagrams can be combined. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a mechanism so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the function / action specified in the flowchart and / or block diagram box.

[0049] These computer program instructions may also be stored in a computer-readable medium, which can instruct a computer, other programmable data processing apparatus, or other device to operate in a specific manner, so that the instructions stored in the computer-readable medium produce an article of manufacture, which includes instructions for implementing the functions / actions specified in the flowchart and / or block diagram blocks.

[0050] The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be executed on the computer, other programmable apparatus, or other devices to produce a computer-implementable program, so that the instructions executed on the computer or other programmable apparatus provide a program for implementing the functions / actions specified in the flowchart and / or block diagram blocks.

[0051] "User interface" used herein is an interface that allows a user or operator to interact with a computer or computer system. "User interface" may also be referred to as "human-machine interface device". A user interface may provide information or data to an operator, and / or receive information or data from an operator. A user interface may enable the input from an operator to be received by a computer, and may provide output to a user from a computer. In other words, a user interface may allow an operator to control or manipulate a computer, and an interface may allow a computer to indicate the effect of the control or manipulation of an operator. Displaying data or information on a display or graphical user interface is an example of providing information to an operator. Receiving data by a keyboard, mouse, trackball, touchpad, pointing stick, graphic input board, joystick, game controller, web camera, head mounted device, pedal, wired gloves, remote controller and accelerometer are all examples of user interface components, and these components enable receiving information or data from an operator.

[0052] As used herein, "hardware interface" encompasses an interface that enables a processor of a computer system to interact with and / or control an external computing device and / or apparatus. A hardware interface may allow a processor to send control signals or instructions to an external computing device and / or apparatus. A hardware interface may also enable a processor to exchange data with an external computing device and / or apparatus. Examples of hardware interfaces include, but are not limited to, a universal serial bus, an IEEE 1394 port, a parallel port, an IEEE 1284 port, a serial port, an RS-232 port, an IEEE-488 port, a Bluetooth connection, a wireless LAN connection, a TCP / IP connection, an Ethernet connection, a control voltage interface, a MIDI interface, an analog input interface, and a digital input interface.

[0053] As used herein, a "display" or "display device" encompasses an output device or user interface suitable for displaying images or data. A display may output visual, auditory, and / or tactile data. Examples of displays include, but are not limited to, computer monitors, television screens, touch screens, tactile electronic displays, Braille screens, cathode ray tubes (CRTs), memory tubes, bi-stable displays, electronic paper, vectorscopes, flat panel displays, vacuum fluorescent displays (VFs), light emitting diode (LED) displays, electroluminescent displays (ELDs), plasma display panels (PDPs), liquid crystal displays (LCDs), organic light emitting diode displays (OLEDs), projectors, and head mounted displays.

[0054] Magnetic resonance imaging (MRI) data is defined herein as the measurements of radio frequency signals emitted by atomic spins recorded using the antenna of a magnetic resonance device during a magnetic resonance imaging scan. A magnetic resonance image or MR image is defined herein as a reconstructed two-dimensional or three-dimensional visualization of anatomical data contained within the magnetic resonance imaging data. The visualization can be performed, for example, using a computer. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In the following, preferred embodiments of the present invention will be described, by way of example only, with reference to the accompanying drawings, in which:

[0056] Figure 1 An example of a magnetic resonance imaging system is illustrated;

[0057] Figure 2 The diagram shows the operation Figure 1 A flowchart of an example of a method of a magnetic resonance imaging system;

[0058] Figure 3 Another example of a magnetic resonance imaging system is illustrated;

[0059] Figure 4 The diagram shows the operation Figure 3 A flowchart of an example of a method of a magnetic resonance imaging system;

[0060] Figure 5 Another example of a magnetic resonance imaging system is shown;

[0061] Figure 6 illustrates an example of an alert dialog box for a graphical user interface;

[0062] Figure 7 illustrates another example of an alert dialog box for a graphical user interface; and

[0063] Figure 8 Yet another example of an alert dialog box for a graphical user interface is illustrated.

[0064] Reference numerals list

[0065] 100 Magnetic Resonance Imaging System

[0066] 104 Magnet

[0067] 106 Hole of magnet

[0068] 108 Imaging Area

[0069] 109 Areas of Interest

[0070] 110 Magnetic field gradient coil

[0071] 112 Magnetic field gradient coil power supply

[0072] 114 Head Coil

[0073] 114' RF Coil

[0074] 116 Transceiver

[0075] 118 Objects

[0076] 119 Facial area

[0077] 120 Object Support

[0078] 122 Camera System

[0079] 122' Camera System

[0080] 122' Camera System

[0081] 123 Multiple Cameras

[0082] 123' Selected Camera

[0083] 124 Mirror

[0084] 125 Object Microphone

[0085] 126 Computer Systems

[0086] 128 Hardware Interface

[0087] 130 processors

[0088] 132 User Interface

[0089] 134 Computer memory

[0090] 140 machine executable instructions

[0091] 142 Pulse sequence commands

[0092] 144 MRI data

[0093] 146 Video images

[0094] 148 Video-based sound activity detection module

[0095] 150 Sound activity status

[0096] 152 Sound Activity Signal

[0097] 200 Controlling the MRI system to acquire MRI data using pulse sequence commands

[0098] 202 Receive video images from the video system

[0099] 204 Determine the sound activity status of the video image in real time by inputting the video image into the video-based sound activity detection module

[0100] 206 If the voice activity state indicates a speaking state, provide a voice activity signal

[0101] 300 Magnetic Resonance Imaging System

[0102] 302 Initial video image

[0103] 304 Audio Data

[0104] 306 Initial video image marked

[0105] 308 Audio-based Voice Activity Detector Module

[0106] 400 Recording an initial video image of the object using a video system

[0107] 402 While recording the initial video image, record audio data using the subject's microphone

[0108] 404 Marking frames as speech or non-speech by feeding the audio data into an audio-based voice activity detector module

[0109] 406 Using frames and per-frame labels to train a video-based sound activity detection module

[0110] 500 Magnetic Resonance Imaging System

[0111] 502 Prompt

[0112] 504 Problem List

[0113] 506 Magnetic Resonance Images

[0114] 508 Speech Recognition Module

[0115] 510 Predefined word lists

[0116] 600 Dialog

[0117] 602 Warning

[0118] 604 Control opening of communication channel

[0119] 700 Dialog Box

[0120] 702 Audio playback controls

[0121] 800 Dialog

[0122] 802 Notify collection to stop DETAILED DESCRIPTION

[0123] Elements with similar reference numbers in the drawings are equivalent elements or perform the same function. Elements that have been discussed previously need not be discussed in subsequent drawings if the function is equivalent.

[0124] Figure 1 An example of a magnetic resonance imaging system 100 with a magnet 104 is shown. The magnet 104 is a superconducting cylindrical magnet with a bore 106 passing through it. Different types of magnets can also be used; for example, a split cylindrical magnet and a so-called open magnet can also be used simultaneously. A split cylindrical magnet is similar to a standard cylindrical magnet, except that the cryostat has been split into two parts to allow access to the isoplane of the magnet, which magnet can be used, for example, in conjunction with charged particle beam therapy. An open magnet has two magnet parts, one above the other, with the space between the two magnets being large enough to accommodate the object: the arrangement of the areas of the two parts is similar to a Helmholtz coil. Open magnets are popular because the object is less constrained. Inside the cryostat of the cylindrical magnet there is a collection of superconducting coils. Within the bore 106 of the cylindrical magnet 104 there is an imaging region 108 in which the magnetic field is strong and uniform enough to perform magnetic resonance imaging. Shown within the imaging region 108 is a region of interest 109. Magnetic resonance data is typically acquired for the region of interest. An object 118 is shown supported by an object support 120 such that at least part of the object 118 is within the imaging region 108 and the region of interest 109.

[0125] There is also a set of magnetic field gradient coils 110 within the bore 106 of the magnet, which is used to acquire initial magnetic resonance data to spatially encode magnetic spins within the imaging region 108 of the magnet 104. The magnetic field gradient coils 110 are connected to a magnetic field gradient coil power supply 112. The magnetic field gradient coils 110 are representative. Typically, the magnetic field gradient coils 110 include three independent sets of coils for spatial encoding in three orthogonal spatial directions. The magnetic field gradient power supply supplies current to the magnetic field gradient coils. The current supplied to the magnetic field gradient coils 110 is controlled according to time and can be tilted or pulsed.

[0126] Adjacent to the imaging region 108 is a radio frequency coil 114 for manipulating the orientation of magnetic spins within the imaging region 108 and for receiving radio transmissions from spins also within the imaging region 108. A head region of a subject 118 is shown within the imaging region 108. In this example, the radio frequency coil is a head coil 114. The head coil 114 is shown surrounding the head of the subject 118. There is a region of interest 109 for imaging a portion of the subject's head. The head coil 114 is also shown to include an optional subject microphone 125 for recording audio data from the subject 118. The audio data may, for example, include speech spoken by the subject 118. There is a camera system 122 on the flange of the magnet 104. The camera 122 may, for example, be referred to as a flange-mounted camera. The camera 122 looks into the bore 106 of the magnet.

[0127] The mirror 124 is positioned so that a reflection of the facial region of the subject 118 is reflected into the camera system 122. The mirror 124 may, for example, be mounted to or incorporated into the head coil 114, or may also be attached to the bore 106 of the magnet 104. Even without the head coil 114, a mirror may be provided for imaging the facial region 119. For example, in other examples, the mirror 124 may be incorporated into a headgear or support that is attached to the subject's head.

[0128] The RF coil 114 or antenna may include multiple coil elements. The RF antenna may also be referred to as a channel or antenna. The RF coil 114 is connected to the RF transceiver 116. The RF coil 114 and the RF transceiver 116 may be replaced by separate transmit and receive coils and separate transmitters and receivers. It should be understood that the RF coil 114 and the RF transceiver 116 are representative. The RF coil 114 is also intended to represent a dedicated transmit antenna and a dedicated receive antenna. Similarly, the transceiver 116 may also represent a separate transmitter and receiver. The RF coil 114 may also have multiple receive / transmit elements, and the RF transceiver 116 may have multiple receive / transmit channels. For example, if a parallel imaging technique such as SENSE is performed, the RF coil 114 will have multiple coil elements.

[0129] The transceiver 116, gradient controller 112, and camera system 122 are shown as being connected to a hardware interface 128 of a computer system 126. The computer system also includes a processor 130 that communicates with the hardware system 128, a memory 134, and a user interface 132. The memory 134 can be any combination of memory accessible to the processor 130. This can include such as main memory, cache memory, and non-volatile memory such as flash RAM, hard drives, or other memory devices. In some examples, the memory 134 can be considered a non-transitory computer readable medium.

[0130] The memory 134 is shown as containing machine executable instructions 140. The machine executable instructions 140 enable the processor 130 to control the operation and functions of the magnetic resonance imaging system 100. The machine executable instructions 140 may also enable the processor 130 to perform various data analysis and computational functions. The computer memory 134 is also shown as containing pulse sequence commands 142. The pulse sequence commands are configured to control the magnetic resonance imaging system 100 to acquire magnetic resonance data from the subject 118 according to a magnetic resonance imaging protocol.

[0131] The pulse sequence commands are commands, or data that can be converted into such commands, that control the various components of the magnetic resonance imaging system 100 to acquire magnetic resonance imaging data 144. The memory 134 is shown as containing magnetic resonance imaging data 144 that has been acquired by controlling the magnetic resonance imaging system 100 using the pulse sequence commands 142. The computer memory 134 is also shown as containing video images 146 that are acquired simultaneously with the magnetic resonance imaging data 144. The video images 146 are input into a video-based voice activity detection module 148 while being acquired.

[0132] For example, video images 146 may be input into the module as they are received by the processor. The video-based voice activity detection module 148 then outputs a voice activity state 150, which may be used to indicate whether the subject 118 is in a speaking state or a non-speaking state. If the subject 118 is in a speaking state, the processor 130 generates a voice activity signal 152. The voice activity signal 152 may be displayed or presented to the operator in different ways. In some cases, an audio signal, a video signal, a visual signal, or a dialog box may be presented on a graphical user interface. In other cases, the voice activity signal 152 is a trigger that causes the processor 130 to execute additional instructions.

[0133] Figure 2 A flow chart is shown, which illustrates a method of operating a magnetic resonance imaging system 100. First, in step 200, the magnetic resonance imaging system 100 is controlled to acquire magnetic resonance imaging data 144 using a pulse sequence command 142. Next, in step 202, while executing the pulse sequence command 142, a video image 146 is received from the camera system 122. This can also be considered as when the magnetic resonance imaging data 144 is being acquired. The two are equivalent. Next, in step 204, a voice activity state 150 is determined in real time for the video image 146 by inputting the video image 146 into a video-based voice activity detection module 148. Finally, in step 206, if the voice activity state 150 indicates a speaking state of the subject 118, a voice activity signal 150 is provided.

[0134] Figure 3 Another example of a magnetic resonance imaging system 300 is illustrated. Figure 3 The examples shown in Figure 1 1, except that camera system 122' has been integrated into head coil 114. Camera system 122' may, for example, be mounted such that when head coil 114 is mounted on the head of subject 118, camera system 122' is directed toward facial region 119.

[0135] The memory 134 is also shown as the acquired initial video image 302. The memory 134 is also shown as containing audio data 304 acquired simultaneously with the initial video image 302. The initial video image 302 and the audio data 304 may be acquired, for example, before the acquisition of the magnetic resonance imaging data, or may be acquired at the beginning of the magnetic resonance imaging protocol in other examples. The memory 134 is also shown as containing a labeled initial video image 306 created by using the audio data 304 as an input to an audio-based voice activity detector module 308. The labeled initial video image 306 may be used, for example, to train the video-based voice activity detection module 148. In some examples, the video-based voice activity detection module 148 may be pre-trained and then the labeled initial video image 306 may be used to adjust the training so that it better responds to the speech of the specific subject 118.

[0136] Figure 4 A flow chart is shown, which illustrates the operation Figure 3 If the magnetic resonance imaging system 100 further comprises an audio-based sound activity detector module 308, then Figure 4 The method shown can also be used to operate Figure 1 A magnetic resonance imaging system 100 is provided.

[0137] First, in step 400, an initial video image 302 is recorded. While the initial video image 302 is being recorded, step 402 is performed. In step 402, audio data 306 of the subject 118 is recorded using the subject microphone 125. Next, in step 404, the frames of the initial video image 302 are labeled by inputting the audio data 304 into the audio-based voice activity detector module 308. The synchronization of the initial video image and the audio data 304 enables the creation of the labeled initial video image 306. Specific frames arranged as frames are labeled as a speaking state or a non-speaking state of the subject 118. These labeled initial video images 306 are then used in step 406, wherein the labeling of each frame is used to train the video-based voice activity detection module 148.

[0138] In some examples, the video-based voice activity detection module is a convolutional neural network. In this case, deep learning can be used to complete the training of the video-based voice activity detection module 148.

[0139] Figure 5 Another example of a magnetic resonance imaging system 500 is illustrated. Figure 5 The magnetic resonance imaging system 500 depicted in FIG. Figure 1 and Figure 3 1. A magnetic resonance imaging system is depicted. In this example, the video system 122" includes a plurality of cameras 123. The plurality of cameras 123 are mounted inside the bore 106 of the magnet 104. For example, compact cameras such as those used in digital devices such as cell phones enable a large number of compact cameras to be placed in the bore 106. This can enable the object 118 to be positioned at a relatively arbitrary position. Figure 1 and Figure 3 In the depicted example, the use of the head coil 114 limits imaging to the head region of the subject 118. In this example, the abdominal region of the subject 118 is imaged. The radio frequency coil 114' is placed on the abdomen of the subject 118 and images the region of interest 109 beneath it. The subject 118 is placed in the bore 106 and then a selected camera 123' can be selected from a plurality of cameras 123 for acquiring images of the facial region 119 of the subject 118. In this example, an optional subject microphone 125 has been incorporated into the subject support 120, which may be in a region near the head of the subject 118. Figure 5 The example shown may be convenient because no special implements or anything need to be placed on the subject 118 .

[0140] Figure 5 Also depicted is an optional prompter 502 that can be used to provide a list of questions 504 to the subject 118. The prompter 502 can, for example, be a video screen that displays the list of questions 504 in text form. If it is a video prompter 502, the prompter can be outside the aperture 106, or even inside. If it is outside the aperture 106, there can be an additional camera for imaging the facial area 118 before the subject enters the magnet 104. In other examples, the prompter 502 can be an audio system for providing the list of questions 504 in audio form. In any case, the prompter 502 generates the questions, the subject 118 responds to the questions, and the questions are used to generate the initial video image 302 and audio data 306.

[0141] The computer memory 134 is also shown as containing a magnetic resonance image 506 reconstructed from the magnetic resonance imaging data 144. The magnetic resonance image 506 may also be present in Figure 1 and Figure 3Similarly, the prompter 502 and the question list 504 can also be incorporated into Figure 1 and Figure 3 In the example shown. In addition, Figure 1 , Figure 3 and Figure 5 The features described in can be combined freely as long as they are not mutually exclusive.

[0142] Figure 5 Also depicted are Figure 1 , Figure 3 and Figure 5 Other optional features in the illustrated example. The memory 134 is also shown as including a speech recognition module 508. During the execution of the pulse sequence command 142 to acquire magnetic resonance imaging data, the audio data 306 can be continuously recorded. The audio data 306 can then be input into the speech recognition module 508 as it is being acquired. Subsequently, the words determined by the speech recognition module 508 can be compared to a predetermined word list 510. For example, the words "burn", "help", "stop" or other words that can indicate that the subject 118 is feeling pain or discomfort can be used by the processor 130 to trigger an immediate response. For example, the detection of a particular word can cause the magnetic resonance imaging system 500 to automatically stop acquiring magnetic resonance imaging data.

[0143] Figure 6 1 shows an example of a dialog box 600 that may be opened on the user interface 132 of the computer system 126 in response to the voice activity signal 152. The dialog box includes a warning 602 that the subject has been detected as speaking. The dialog box 600 also includes a control 604 that enables an operator to open a communication channel with the subject 118. For example, this may enable the subject 118 to use a Figure 1 , Figure 3 and Figure 5 The object microphone 125 depicted in FIG. 1 speaks to the operator, and there may then also be a speaker or audio system for providing audio from the operator. In other examples, the control 604 is a manual control that is not incorporated into the computer system 126.

[0144] Figure 7 A more complex example of a dialog box 700 that can be generated on a user interface in response to a voice activity signal 152 is shown. Figure 7 The dialog box 700 contains Figure 6702, but with the addition of an audio playback control 702. In this example, the audio playback control 702 can be used to play back the portion of the audio data 306 corresponding to the time when the subject 118 was detected to be speaking. This can, for example, enable the operator to see what the subject 118 said when the speaking state was detected. For example, if the subject 118 is only talking to himself or herself or saying something unrelated to the magnetic resonance imaging scan, it can help prevent the operator from stopping the execution of the pulse sequence command when it is not necessary.

[0145] Figure 8 Yet another example of a dialog box 800 is shown. In this example, there is also an alert 602 that subject speech has been detected. However, in this example, there are no additional controls, but rather a notification 802 that acquisition has stopped. This can be, for example, a dialog box 800 generated in response to recognizing that the subject 118 has spoken a word that matches a word on the predetermined word list 510.

[0146] As mentioned above, currently the subject uses an old-fashioned pressure-operated balloon switch to communicate that he wishes to communicate during the scan because the patient intercom (communication system) is turned off due to scanner noise.

[0147] There is a recent research trend to use an in-bore camera to detect gross patient motion and physiological movement during scanning. An example could be to use such a camera to detect typical mouth movements when speaking, which has demonstrated application as microphone control in audio conferencing. This technology can be used to trigger (provide a sound activity signal) events that are usually triggered by the patient pressing a balloon switch (squeezing the bulb), and can therefore be omitted. Thus, time is saved in setting up the switch and instructing the patient. The method also avoids the need for patient comprehension and compliance, which is not suitable for children, patients with dementia, and patients with mental disabilities. Likewise, the patient no longer needs to hold the balloon uncomfortably in his hand, and if he drops the balloon, he can retrieve it without moving.

[0148] The paradigm can overcome one or more of the following disadvantages of using balloon switches. Using balloon switches has the following disadvantages:

[0149] -During the scan, the patient must hold the balloon in his hand, which may become uncomfortable

[0150] - The patient may drop the balloon from his hand and move at least his hand and arm to retrieve it, which may introduce motion artifacts.

[0151] -The MR operator must set the switch for each patient, which takes time.

[0152] -The MR operator must instruct the patient on how to use the switch, which takes time.

[0153] - Patients must understand and follow instructions, which may be difficult for some patients, such as children, patients with dementia, and patients with mental disabilities.

[0154] -Balloons will increase the bill of materials for the scanner. The software used for the examples can be left without it.

[0155] Some examples may use an in-bore camera to detect typical speaking mouth movements (facial movements). This is used to trigger an event that is usually triggered by the patient pressing the balloon, so this can be omitted.

[0156] Some of these techniques for detecting speech in video applications may be based on neural networks that require algorithm training. Some examples may use a typical first communication between an MR operator and a subject for this training or may tailor the training to a specific subject.

[0157] The paradigm can generate training data by integrating automatic patient communication, starting from when the RF gate is closed after positioning the patient, until the silent period of the preparation phase for the first scan. This communication drives the subject to speak and the corresponding synchronized video and audio data are used to train the video analysis. The concept additionally saves time and relieves the operator from routine tasks when asking the patient if he is comfortable and enables the operator to be clearly understood when the operator is very busy preparing for the first scan. The possibility for the MR operator to interactively jump in itself can be combined with the use of pre-recorded audio clips with the current MR operator's voice to avoid the patient noticing that the conversation is performed by a computer.

[0158] The method can use video analysis (using a video-based voice activity detector) to determine whether the patient is currently speaking or not speaking. Most known solutions for such video analysis are based on neural networks and need to be trained using suitable video data.

[0159] Video analysis can have real-time capabilities, and the computation time for training must be kept short enough to be performed before the first scan begins. Suitable training data that marks whether the patient is speaking or not is not easily available. Speech detection algorithms that run without appropriate training data are more prone to errors. For safety reasons, the patient alarm mechanism should have a very low false negative rate (cases where a patient alarm is not detected), while for reasons of high scanning efficiency, the patient alarm mechanism should have a low false positive rate (cases where a patient alarm is false).

[0160] In one example, a camera and microphone can be integrated into the bore of an MRI magnet. Automatic patient-operator / scanner communication is performed during the preparation phase of the first scan and between additional scans. This communication prompts the patient to speak and trains the video analysis using the output of synchronized video and audio data.

[0161] In this example, at least one camera is built into the bore of the scanner so that it can see the patient's face in the most common scanning applications and patient positions. A suitable location is at the top of the bore, offset about 20 cm beyond the isocenter toward the rear of the magnet, since most patients are scanned in the supine position, with the face approximately in this position. A second camera is also advantageous, also at the top of the bore, but with a different offset along the B0 field axis so that at least one of the cameras can see the face in a range of common positions. A microphone is added to each of the at least one cameras.

[0162] The training data for video analysis can include short clips of about 0.5s duration in which the patient is either speaking or not speaking. Each segment can be marked with an almost negligible error rate, respectively, because incorrect marking will destroy the training data and lead to an increased error rate for real-time video analysis, which is prohibited. In order to generate such robust labeled training data, it is proposed to make the patient speak when the scanner noise is almost non-existent. At that time, a relatively simple audio analysis can be performed to detect the speaking time period and to perform robustly. The video data is divided into speaking / not speaking segments (speaking state / not speaking state) using this information. Voice activity detection (VAD) algorithms are known in speech recognition to detect speaking time periods from audio data. For example, these labeled segments can be used to train a neural network.

[0163] Training Process

[0164] Computer-controlled communication can be used to automate the communication with the patient to generate the training data. This saves time and relieves the operator from performing this task during a period that is usually very busy preparing for the first scan. This communication can be started as soon as the RF gate is closed after positioning the patient, and finally the silent period of the preparation phase of the first scan is used. The MR operator can hear this communication so that he can intervene in the case of any early patient requests or alarms. During this communication, audio clips with the individual MR operator's own voice can be pre-recorded and played. In the case of more complex requests from the patient, the MR operator can jump into the communication interactively. This can avoid the patient noticing that he is talking to the machine.

[0165] Typical communication:

[0166] 1. The scanner (MRI system) asks: "Can you hear me clearly?" The patient can answer, and the speech recognition system can analyze the answer as positive or negative. In the case of a negative answer, the patient is informed that the volume of the speaker will increase. In the case of a positive answer, the scanner can ask: "Is the volume comfortable for you?" This dialogue can be looped until the appropriate volume is reached.

[0167] 2. The scanner can ask for the patient’s name and age: “I want to make sure I understand you clearly. So, could you please state your full name and age?”

[0168] 3. Additional questions can be used to ask about the patient’s comfort (position, temperature, and similar aspects). The voice recognition system can even evaluate the answers and adjust the temperature.

[0169] Known typical scanner noise

[0170] Most of the preparation phases for the first scan are either very quiet or produce low noise consisting of short clicks, which is very reproducible (e.g., resonant frequency determination). Such time periods can also be used for training data generation. It is therefore proposed to train the VAD algorithm to ignore this known noise.

[0171] Scanning Process

[0172] During a normal scan, machine learning-based video analytics are used to detect when a patient starts talking. In this case, the MR operator's speaker is unmuted. The operator can then decide to terminate the scan.

[0173] While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the invention is not limited to the disclosed embodiments.

[0174] By studying the drawings, the disclosure and the appended claims, those skilled in the art will be able to understand and implement other variations of the disclosed embodiments when practicing the claimed invention. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit may implement the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not mean that a combination of these measures cannot be used to advantage. The computer program may be stored / distributed on an appropriate medium, such as an optical storage medium or solid-state medium provided with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems. Any reference numerals in the claims should not be interpreted as limiting the scope of protection.

Claims

1. A magnetic resonance imaging system (100, 300, 500) configured to acquire magnetic resonance imaging data (144) from a subject (118) at least partially within an imaging region (108), wherein: The magnetic resonance imaging system comprises: a video system (122, 122', 122") configured to provide a video image (146) of a facial region (119) of the subject during acquisition of the magnetic resonance imaging data; - a subject microphone (125) configured to acquire audio data (304) from the subject; - a memory (134) for storing machine executable instructions (140) and pulse sequence commands (142), wherein the pulse sequence commands are configured to control the magnetic resonance imaging system to acquire the magnetic resonance imaging data; - a processor (130) for controlling the magnetic resonance imaging system, wherein execution of the machine executable instructions causes the processor to: - recording (400) an initial video image (302) of the object using the video system, wherein the initial video image comprises frames; - recording (402) said audio data using said subject microphone while recording said initial video image; - marking (404) the frame as speech or non-speech by inputting the audio data into an audio-based voice activity detector module (308); and - using the frames and the labels of each of the frames to train (406) a video-based voice activity detection module to determine (204) a voice activity state (150) in real time from the video images; - controlling (200) the magnetic resonance imaging system to acquire the magnetic resonance imaging data using the pulse sequence command; - receiving (202) the video image from the video system during execution of the pulse sequence command; - determining (204) a voice activity state (150) in real time based on the video image by the video-based voice activity detection module having access to the video image, wherein the voice activity state indicates a speaking state of the subject or a non-speaking state of the subject; - if the voice activity state indicates the speaking state, providing (206) a voice activity signal (152), wherein - execution of the machine executable instructions further causes the processor to provide an indicator (602) on a user interface as the voice activity signal, and - the user interface is configured for opening a communication channel between an operator and the subject in response to the voice activity signal.

2. The magnetic resonance imaging system according to claim 1, wherein: The audio-based voice activity detector module is configured to exclude gradient coil noise from the training (406) of the video-based voice activity detection module during labeling of the frames as speech or non-speech.

3. The magnetic resonance imaging system according to claim 1 or 2, wherein: The magnetic resonance imaging system also includes a prompter (502) configured to present questions to the subject, wherein execution of the machine-executable instructions further causes the processor to present a list of questions (504) to the subject using the prompter during recording of the initial video image and initial audio.

4. The magnetic resonance imaging system according to any one of claims 1 to 2, wherein: The magnetic resonance imaging system further comprises a subject support (120) configured for supporting at least a portion of the subject in the imaging region, wherein the subject microphone is incorporated in the subject support.

5. The magnetic resonance imaging system according to any one of claims 1 to 2, wherein: Execution of the machine executable instructions further causes the processor to: - continuously recording the audio data using the subject microphone during acquisition of the magnetic resonance imaging data, wherein the audio data is synchronized with the video images; - marking the audio data with the voice activity state; and - upon receiving a signal from a user interface (700), playing back said audio data marked with said speaking state.

6. The magnetic resonance imaging system according to any one of claims 1 to 2, wherein: Execution of the machine executable instructions further causes the processor to: - receiving a spoken word list by inputting said audio data into a speech recognition module (508); - performing any of the following operations: issuing a predetermined alarm in a user interface of the magnetic resonance imaging system if at least part of the spoken word list matches a predetermined word list (510); stopping the acquisition of the magnetic resonance imaging data if at least part of the spoken word list matches the predetermined word list; and combinations thereof.

7. A magnetic resonance imaging system according to any one of the preceding claims, wherein: The video-based voice activity detection module includes a convolutional neural network.

8. A magnetic resonance imaging system according to any one of the preceding claims, in, The magnetic resonance imaging system comprises a magnet having a bore (106), wherein the camera system comprises a plurality of cameras (122") mounted within the bore.

9. A magnetic resonance imaging system according to any one of the preceding claims, wherein: The magnetic resonance imaging system comprises a head coil (114), wherein the camera system comprises a coil-mounted camera (122') attached to the head coil.

10. A magnetic resonance imaging system according to any one of the preceding claims, wherein: The camera system includes a flange-mounted camera (122) configured to image a reflected image of a facial region of the subject.

11. A method of operating a magnetic resonance imaging system (100, 300, 500), the magnetic resonance imaging system being configured to acquire magnetic resonance imaging data (144) from a subject (118) at least partially within an imaging region, wherein: The magnetic resonance imaging system comprises a video system (122, 122', 122"), the video system being configured to provide a video image (146) of a facial region (119) of the subject during acquisition of the magnetic resonance imaging data, and a subject microphone (125) being configured to acquire audio data (304) from the subject, wherein the method comprises: - recording (400) an initial video image (302) of the object using the video system, wherein the initial video image comprises frames; - recording (402) said audio data using said subject microphone while recording said initial video image; - marking (404) the frame as speech or non-speech by inputting the audio data into an audio-based voice activity detector module (308); and - using the frames and the labels of each of the frames to train (406) a video-based voice activity detection module to determine (204) a voice activity state (150) in real time from the video images; - controlling (200) the magnetic resonance imaging system to acquire the magnetic resonance imaging data using a pulse sequence command (142), wherein the pulse sequence command is configured to control the magnetic resonance imaging system to acquire the magnetic resonance imaging data; - receiving (202) the video image from the video system during execution of the pulse sequence command; - determining (204) a voice activity state (150) for the video image in real time by inputting the video image into the video-based voice activity detection module (148), wherein the voice activity state indicates a speaking state of the subject or a non-speaking state of the subject; and - if the voice activity state indicates the speaking state, providing (206) a voice activity signal (152), wherein - providing an indicator (602) on a user interface as said voice activity signal, and - the user interface is configured for opening a communication channel between an operator and the subject in response to the voice activity signal.

12. A computer program product comprising machine executable instructions (140) for execution by a processor (130) controlling a magnetic resonance imaging system, the magnetic resonance imaging system being configured to acquire magnetic resonance imaging data from a subject (118) at least partially within an imaging region, wherein: Execution of the machine executable instructions causes the processor to: - recording (400) an initial video image (302) of the object, wherein the initial video image comprises frames; - recording (402) audio data (304) from the subject simultaneously with recording the initial video image; - marking (404) the frame as speech or non-speech by inputting the audio data into an audio-based voice activity detector module (308); and - using the frames and the labels of each of the frames to train (406) a video-based voice activity detection module to determine (204) a voice activity state (150) in real time from the video images; - receiving (202) a video image from a video system (122, 122', 122") during the acquisition (200) of the magnetic resonance imaging data; - determining (204) a voice activity state (150) in real time for the video image by inputting the video image into the video-based voice activity detection module (148), wherein the voice activity state indicates a speaking state of the subject or a non-speaking state of the subject; and - providing (206) a voice activity signal (152) if the voice activity state indicates the speaking state, and - providing an indicator (602) on a user interface as said voice activity signal, and - opening a communication channel between an operator and the subject in response to the voice activity signal.

Images