Detection of safety conditions during medical

Abstract

A medical system (100, 300) is disclosed herein. Execution of the machine executable instructions (120) causes the computing system (104) to: receive (200) an overview image (128); receiving (202) an object locator (130) in response to inputting the overview image into the first neural network (122); constructing (204) an object sub-image (132) using the object locator; receiving (206) object position data (134) in response to inputting the object sub-image of each of the plurality of predetermined objects into the object-specific neural network (124); constructing (208) a plurality of search images (136) using the object position data of the plurality of predetermined objects; receiving (210) locator data (138) describing a position of the object in response to inputting each of the plurality of search images into the object-specific neural network (126); detecting (212) a security state (140) by comparing the locator data and the object position data of each of the predetermined objects with respect to predetermined criteria; and providing (214) a warning signal (144) in the event that the safety state is detected.

Description

Technical Field

[0001] The present invention relates to medical imaging and, more particularly, to safety during medical imaging procedures. Background Art

[0002] Various medical imaging techniques (e.g., magnetic resonance imaging (MRI), computed tomography, positron emission tomography, and single photon emission tomography) enable detailed visualization of a subject's anatomical structures. In medical imaging modalities such as these, it is critical to properly position the subject prior to the procedure. Otherwise, the subject may be incorrectly imaged.

[0003] U.S. patent application publication US20213 / 012659A1 discloses a medical device, comprising a medical imaging system configured to acquire medical imaging data from a subject, a subject support configured to support the subject during the acquisition of the medical imaging data, and an optical imaging system configured to acquire optical imaging data of the subject on the subject support. The execution of the machine executable instructions causes the processor controlling the medical device to perform the following: control the optical imaging system to acquire optical imaging data, generate an initial vector using the optical imaging data, generate a synthetic image by inputting the initial vector into a generator neural network, calculate the difference between the synthetic image and the optical imaging data, and provide a warning signal if the difference differs by a predetermined threshold. The generator neural network is trained to generate a synthetic image of the subject on the subject support in response to the input initial vector.

[0004] U.S. patent application publication US2021 / 0080531 A1 discloses using a deep learning neural network to determine scan settings for a localizer scan, including mapping an MR calibration image to a corresponding anatomical region of interest.

[0005] U.S. patent application publication US2022 / 0189622 A1 discloses a camera system that uses a camera system and a neural network to identify misplaced objects associated with a medical device. Summary of the invention

[0006] The invention provides a medical system, a computer program and a computer-implemented method in the independent claims.

[0007] Using camera systems to monitor the setup and safety of medical imaging systems can present some challenges. One challenge is that if the system has even a small error rate, the operator may disable it. To be useful in clinical situations, such a system needs to have near-perfect accuracy. Another difficulty is that properly training the neural networks used to monitor images can be very labor-intensive.

[0008] Embodiments may have the advantage of high accuracy and the ability to be easily adjusted to specific clinical situations of medical imaging protocols. Embodiments may achieve this by using a hierarchy of neural networks, such as described below: First, an overview image is received. A first neural network first provides object locators to identify a series of predetermined objects in a lower resolution version of the overview image. These object locators are then used to build object sub-images from the overview image, which are then input into object-specific neural networks. The object sub-images have a medium resolution. The object locators may then have additional information for building multiple search images from the overview image. Specialized object-specific neural networks are then used to search for specific objects or safety conditions in the multiple search images. Using a hierarchy of neural networks can provide neural networks that can be accurately trained for specific tasks. The hierarchy can also enable object-specific neural networks and object-specific neural networks to be easily added to or removed from the system.

[0009] In one aspect, the present invention provides a medical system comprising a memory storing machine executable instructions. The memory also stores a first neural network, a subject-specific neural network, and an object-specific neural network. The medical system also includes a computing system. Execution of the machine executable instructions causes the computing system to receive an overview image. The overview image has a native resolution and describes a subject position on a subject support of a medical imaging system. Execution of the machine executable instructions also causes the computing system to receive an object locator describing the position and orientation of a plurality of predetermined objects in the overview image in response to inputting the overview image into the first neural network.

[0010] The first neural network is configured to process the overview image at a first resolution. The first resolution is lower than the original resolution. The plurality of predetermined objects include a subject and a subject support. Execution of the machine-executable instructions further causes the computing system to construct an object sub-image for each of the plurality of predetermined objects using the object locator. Each object sub-image has a second resolution. The second resolution is lower than the original resolution and higher than the first resolution. Execution of the machine-executable instructions further causes the computing system to receive object position data describing each of the plurality of predetermined objects in response to inputting the object sub-image of each of the plurality of predetermined objects into the object-specific neural network.

[0011] The object-specific neural network is unique to each of a plurality of predetermined objects. Execution of the machine-executable instructions further causes the computing system to construct a plurality of search images using the object position data of the plurality of predetermined objects. The plurality of search images have a third resolution that is higher than the second resolution. The third resolution is less than or equal to the original resolution. Execution of the machine-executable instructions further causes the computing system to receive locator data describing the location of the object in response to inputting each of the plurality of search images into the object-specific neural network. Execution of the machine-executable instructions further causes the computing system to detect a safety state by comparing the locator data and the object position data of each of the predetermined objects relative to a predetermined standard. Execution of the machine-executable instructions further causes the computing system to provide a warning signal if a safety state is detected.

[0012] The safety state can be, for example, a flag or label in the software that is set when predetermined criteria are met. In one specific example, multiple search images can define zones or areas that are unfavorable for a subject to place his or her hands. Multiple search images can be searched using an object-specific neural network that is specially trained to locate hands. When the object-specific neural network detects a hand, a safety state will be set using the object-specific neural network, which indicates that the specific search image has a hand located therein. The warning signal can take different forms in different examples. For example, it can cause a graphical user interface to display a warning or additional information. In other cases, the warning signal can be used to change the behavior of the medical system. For example, it can cause the medical system to suspend operation until an operator issues a command. In other cases, it can disable various functions, such as moving a subject support. The warning signal can be integrated into a control loop or control system.

[0013] Multiple embodiments may be beneficial because there is a hierarchy of neural networks provided. For example, using the previous example, it may be difficult to train recognition of a hand in a position that may be dangerous to the subject using a single neural network. The first neural network runs at a lower resolution and is able to recognize general objects or label them. This is a well-known task and various neural networks can perform well. The neural network can also be trained by making general labels in the image. The next level is the object-specific neural network.

[0014] This can actually be a neural network or a combination of neural networks working together so that each object is viewed specifically using a neural network trained for that specific object. This is then used to provide detailed information about the orientation or object position data. The object position data can then be used to define the positions of multiple search images. Sometimes, multiple search images can be defined in terms of object position data from one object, or the object position data can come from multiple objects. For example, in one example, object position data from two objects can be used to trace a line extending between two objects or to ensure that they have a certain amount of distance between each other. In the case of a subject support, multiple search images can be defined using the positions on a specific subject support.

[0015] The various image resolutions of the hierarchy of neural networks can be provided in a variety of ways. For example, there may be an intermediate step where the resolution of the various images is reduced by performing an averaging process. In other examples, the neural network may perform this operation on its own using an average max pooling layer at the input. That is, for example, a first neural network may have an average max pooling layer that reduces the overview image from the original resolution to the first resolution.

[0016] The overview image may take different forms in different examples. In one example, the overview image is a conventional image from a camera or video system. The conventional image may be, for example, an RGB or grayscale image. In this case, the conversion between the first resolution, the second resolution, and the third resolution involves a conversion or reduction in the resolution of the overview image when constructing the image that is input into the neural network. As described above, the resolution can be reduced when reconstructing the image, or it can be done by a pooling layer at the input of one of the neural networks.

[0017] Another possibility is that the overview image contains composite data from multiple image modalities or sensor types. For example, a camera or video system may provide a 3D image as well as an RBG or grayscale image. In this case, the image will have more than three data channels. It will have three RGB channels plus an additional channel with depth data. In this case, the neural network will be configured to accept 4 data channels instead of three input channels. The reduction of resolution to a first resolution, a second resolution and a third resolution is understood to apply to all channels of the overview image.

[0018] As described above, the object-specific neural network may be a single neural network or a series of neural networks. In another embodiment, multiple search images may be defined by dedicated specific system risk regions. These dedicated specific system risk regions may be defined in terms of relative coordinates of object position data of specific objects.

[0019] In some specific examples, the warning signal can be used to stop the automatic settings of the medical imaging system. For example, when the finger is in an unsafe position or posture, the movement of the subject support can be stopped. Another example is to operate the examination table or subject support when a collision is likely. For example, if a collision is likely, the examination table or subject support can be operated to have a speed that is less likely to injure someone.

[0020] In another example, a warning signal may cause a medical system to adjust its behavior so that it only allows manual overrides through controls that are close to the patient. For example, there may be controls for inserting a subject into a medical imaging system. The warning signal may force or modify the system so that only controls that are adjacent to the patient or subject are allowed to move the subject. This would reduce the chances of a remote operator causing harm to the subject.

[0021] One advantage of using a hierarchy of neural networks is that specialized neural networks are more specific and sensitive for a given capacity. This can have the benefit of increasing the accuracy of the overall system and reducing the occurrence of false positives. Moreover, the neural networks trained separately for each target are often complementary. Another advantage is that additional objects and things can be added to the system without modifying the detection performance of the existing object detection network. The first neural network will be trained to recognize all objects currently being used, but below that, the specific neural networks in the hierarchy, once trained, will not need to be retrained.

[0022] The neural networks used in this medical system (including the first neural network, the object-specific neural network, and the object-specific neural network) can be implemented in different architectures. Example architectures of neural networks that can be used can include elements of ResNet, U-net, and general full convolutional layers. Such full convolutional neural networks can be particularly beneficial because they are particularly suitable for detecting and locating a single object against the background of many other objects, i.e., have shown very good sensitivity and specificity.

[0023] A first neural network may be trained with images having a first resolution that have been labeled with a plurality of predetermined objects (labeling may include outlines, segmentation masks, keypoints, or bounding boxes). An object-specific neural network may be trained with images having a second resolution that have been labeled with position data of a specific object to which the object-specific neural network is assigned. An object-specific neural network may be trained with images having a third resolution that have been trained with a specific object.

[0024] In another embodiment, the use of predetermined criteria can be used to provide a hierarchy of safety hazards and risks. For example, the predetermined criteria can provide a ranking of the severity of events. For example, there can be multiple safety hazard categories, such as normal safety hazards and high severity safety hazards. The basic response of the medical system can adjust its function according to these two categories.

[0025] In some examples, the warning signal can be displayed or provided to the user via a visual user interface or other audible system. The warning signal can be used, for example, to control or limit the use of the machine, for example, if a dangerous situation exists, the medical system can be disabled or temporarily disabled. Each finding or safety risk identified by the system can, for example, be assigned a different type or level of warning. As mentioned above, the warning signal can also be a control signal for the medical system.

[0026] In another embodiment, in the event that a warning signal is provided, execution of the machine executable instructions also causes the computing system to display a warning graphical user interface. The warning graphical user interface is configured to perform any of the following: provide spatial localization of the cause of the safety state, display the warning signal type and / or warning context for use using predetermined criteria, display a severity ranking using predetermined criteria, display operator instructions retrieved from an instruction database in response to a query using predetermined criteria, provide a voice or audible alarm, project a local warning light onto the subject using a projector, and combinations of the above. When the local warning light is configured to (in some examples) illuminate an object that triggers the safety state, for example, if the subject's hand is at risk of being squeezed or pinched, the light can be directed onto the hand, which can provide an improved way to ensure that the subject and / or the operator places the subject's hand in a safer position.

[0027] In another embodiment, an object-specific neural network for a subject is configured to output a set of anatomical keypoint locators. At least a portion of a plurality of search images is determined using a predetermined geometric relationship or position relative to the set of anatomical keypoint locators. For example, in the case of testing when a subject places his or her hand in a position where the subject may cause a conductive loop, various anatomical locations where the subject may place the hand in an unfavorable position may be defined relative to the set of anatomical keypoint locators. For example, the locations of the plurality of search images may be defined in terms of coordinates of the set of anatomical keypoint locators. Once the set of anatomical keypoint locators are determined from the object-specific neural network, the locations of the plurality of search images may be calculated using these locators.

[0028] Anatomical key points as used herein encompass anatomical landmarks or locations within a subject. Anatomical key point locators are often referred to as joint locations. However, anatomical key points are broader than the locations of joints. Anatomical key points may include the locations of various joints or anatomical parts of a subject. A camera image describes a subject and provides a description of the exterior or surface of the subject. For example, a camera image may be an optical image, an infrared image, or even a color image. In other examples, the camera image is a three-dimensional surface image.

[0029] In another embodiment, the execution of the machine executable instructions also causes the computing system to filter the locator data using a locator filter adjusted by the key point locator. The execution of the machine executable instructions also causes the computing system to ignore the locator filter data filtered by the locator filter. In this embodiment, the system can be programmed to ignore the locator data defined by the specific key point locator. This may be very useful, for example, when ignoring data from an operator rather than a subject. For example, a subject may be lying on a subject support or an examination table, and the operator may be configuring the medical system. Then, the data related to the operator can be ignored or processed separately.

[0030] In another embodiment, the object is a nurse call squeeze ball. Using a specialized neural network to identify this object may be beneficial because the nurse call squeeze ball is often located in the subject's hand and may otherwise be difficult to locate using a general neural network.

[0031] In another embodiment, the object is a headset and / or earplug. The plurality of search images include a head region of the subject. The locator data describes the position of the object relative to a central locator of the subject's ear.

[0032] In another embodiment, the object is a conductive cable. Multiple predetermined objects include magnetic resonance imaging coils and / or wired devices. The wired device in this article is a device connected to a control line or a communication line. Multiple predetermined objects also include at least one magnetic resonance transmitting body coil element. Multiple search images include areas adjacent to the magnetic resonance imaging coil and / or wired device. Multiple search images also include areas adjacent to at least one transmitting body coil element. Multiple search areas also include additional areas having predetermined positions relative to the set of anatomical key point locators. This embodiment may be beneficial because it can keep cables leading to the magnetic resonance imaging coil or wired device away from locations that may ultimately cause radio frequency burns to the subject.

[0033] In another embodiment, the object is a hand. The warning signal is configured to provide a dangerous hand position warning signal in the event that the hand is in a pinched position or a current loop induced position. As used herein, a current loop induced position encompasses a position where a subject places his or her hand so that a conductive loop is formed through the subject's body and arm. During a magnetic resonance imaging examination, this may result in radio frequency field induced currents or locally increased SAR exposure, resulting in heating and / or burns to the subject.

[0034] Multiple search images can be selected to view specific dangerous or high risk locations. They can use the recognition results of the body model and / or other objects to identify high resolution areas. The search definition area or multiple search images can be defined in terms of whether placing the hand on the subject support is dangerous or related to anatomical structures such as the hip area.

[0035] In another embodiment, the object is a magnetic resonance condition device, such as a device that allows for a specific set of magnetic resonance imaging scheme parameters or a specific position relative to the MRI system. The position of the magnetic resonance condition device can be derived from the specified and clear use case of the specific magnetic resonance imaging scheme. This can include things such as keeping the device within a certain strength of the B0 field, and searching for images so as to search for images covering certain magnetic contours. In other cases, this can involve ensuring that a certain distance between different devices is maintained, and the search image can cover the problem area between or between these devices.

[0036] In another embodiment, the plurality of predetermined objects include one or more magnetic resonance imaging coils. The locator data includes radio frequency cable position data. Execution of the machine executable instructions further causes the computing system to determine the safety state by performing any of the following: comparing the object position data and the radio frequency position data of the one or more magnetic resonance imaging coils with a set of allowable radio frequency routing paths; comparing the object position data of the one or more magnetic resonance imaging coils with a set of reported magnetic resonance imaging coil connections; and combinations of the foregoing. The reported magnetic resonance imaging coil connections may, for example, be provided by a radio frequency system of the magnetic resonance imaging system, the radio frequency system being capable of detecting whether the coil is attached.

[0037] In another embodiment, the overview image is an optical image.

[0038] In another embodiment, the overview image is a color optical image.

[0039] In another embodiment, the overview image is a three-dimensional image. For example, a so-called three-dimensional camera can be used to acquire the three-dimensional image.

[0040] In another embodiment, the medical system further comprises a medical imaging system having a subject support. The medical system further comprises an imaging device configured to acquire an overall image by imaging at least the subject support. Execution of the machine executable instructions further causes the computing system to control the imaging device to acquire an overview image.

[0041] In another embodiment, the medical imaging system is a magnetic resonance imaging system.

[0042] In another embodiment, the medical imaging system is a computed tomography system.

[0043] In another embodiment, the medical imaging system is a combined magnetic resonance imaging and computed tomography system.

[0044] In another embodiment, the medical imaging system is a positron emission tomography system.

[0045] In another embodiment, the medical imaging system is a combined magnetic resonance imaging system and computed tomography system.

[0046] In another embodiment, the medical imaging system is a single photon emission tomography system.

[0047] In another embodiment, the medical imaging system is a digital X-ray system.

[0048] In another embodiment, the medical imaging system is a digital fluoroscopy system.

[0049] In another embodiment, the medical imaging system is a magnetic resonance imaging guided radiation therapy system.

[0050] In another embodiment, the medical imaging system is a computed tomography guided radiation therapy system.

[0051] In another embodiment, the medical imaging system is a magnetic resonance imaging guided high intensity focused ultrasound system.

[0052] In another embodiment, the medical imaging device is a three-dimensional camera.

[0053] In another embodiment, execution of the machine executable instructions further causes the computing system to compare the spatial profile of the object to a predetermined volume in order to detect a collision condition.

[0054] In another embodiment, execution of the machine executable instructions further causes the computing system to use the contour data from the three-dimensional camera to filter out and ignore localizer data that is not relevant to the examination setting. For example, the three-dimensional camera can be used to determine or identify objects that are outside the area of ​​the subject support, i.e., objects that are not affected by the subject transport mechanism (equipment, operator).

[0055] In another embodiment, execution of the machine executable instructions further causes the computing system to determine a signal motion correlation associated with subject support motion so as to filter out and ignore localizer data that is not relevant to the examination setting. For example, when the subject support is moved, portions of the image that are moved may be detected in the three-dimensional image. Regions outside of the moved region may then be identified as not relevant to the examination setting.

[0056] In another embodiment, the plurality of predetermined objects include an operator. Operator as used herein encompasses a person who operates or controls a medical system. The execution of the machine executable instructions also causes the computing system to use the operator's object locator to identify the operator region in the overview image. The execution of the machine executable instructions also causes the computing system to separate the locator data and / or object position data within the operator region. For example, this may be useful in the case of searching for the subject's anatomical structure but wishing to ignore the operator's anatomical structure. For example, when determining whether a subject is at risk of having his or her fingers pinched, one may wish to ignore the position of the operator's hand.

[0057] In another embodiment, the overview image is received as an image stream having a frame rate. The first neural network is configured to process the overview image at a rate equal to or greater than the frame rate. Execution of the machine executable instructions further causes the computing system to repeatedly input the overview image to obtain the object locator at the frame rate. In this embodiment, the first neural network operates at a sufficiently low resolution that the image can be processed at a rate equal to or greater than the frame rate. This means that the first neural network is essentially able to recognize objects in real time.

[0058] In another aspect, the invention provides a computer program comprising machine executable instructions, a first neural network, a subject specific neural network, and an object specific neural network. Execution of the machine executable instructions causes a computing system to receive an overview image. The overview image has a native resolution and describes a subject position on a subject support of a medical imaging system. Execution of the machine executable instructions also causes the computing system to receive an object locator describing a position and orientation of a plurality of predetermined objects in the overview image in response to inputting the overview image into the first neural network. The first neural network is configured to process the overview image at a first resolution. The first resolution is lower than the native resolution. The plurality of predetermined objects include a subject and a subject support.

[0059] Execution of the machine executable instructions further causes the computing system to construct an object sub-image for each of the plurality of predetermined objects using the object locator. Each object sub-image has a second resolution. The second resolution is lower than the original resolution and higher than the first resolution. Execution of the machine executable instructions further causes the computing system to receive object position data describing each of the plurality of predetermined objects in response to inputting the object sub-image of each of the plurality of predetermined objects into an object-specific neural network that is unique to each of the plurality of predetermined objects.

[0060] Execution of the machine executable instructions also causes the computing system to construct a plurality of search images using object position data of a plurality of predetermined objects. The plurality of search images have a third resolution higher than the second resolution. The third resolution is less than or equal to the original resolution. Execution of the machine executable instructions also causes the computing system to receive locator data describing the location of the object in response to inputting each of the plurality of search images into an object-specific neural network. Execution of the machine executable instructions also causes the computing system to determine a safety state by comparing the locator data and the object position data of each of the predetermined objects relative to a predetermined standard. Execution of the machine executable instructions also causes the computing system to provide a warning signal if a safety state is detected.

[0061] In another aspect, the present invention provides a computer-implemented method. The method includes receiving an overview image. The overview image has a native resolution and describes a subject position on a subject support of a medical imaging system. The method also includes receiving an object locator describing the position and orientation of a plurality of predetermined objects in the overview image in response to inputting the overview image into a first neural network. The first neural network is configured to process the overview image at a first resolution. The first resolution is lower than the native resolution. The plurality of predetermined objects include a subject and a subject support.

[0062] The method also includes constructing an object sub-image for each of the plurality of predetermined objects using the object locator. Each object sub-image has a second resolution. The second resolution is lower than the original resolution and higher than the first resolution. The method also includes receiving object location data describing each of the plurality of predetermined objects in response to inputting the object sub-image of each of the plurality of predetermined objects into an object-specific neural network that is unique to each of the plurality of predetermined objects. The method also includes constructing a plurality of search images using the object location data for the plurality of predetermined objects.

[0063] The plurality of search images have a third resolution higher than the second resolution. The third resolution is less than or equal to the original resolution. The method also includes receiving locator data describing the location of the object in response to inputting each of the plurality of search images into an object-specific neural network. The method also includes detecting a safety condition by comparing the locator data and the object location data for each of the predetermined objects relative to a predetermined standard. The method also includes providing a warning signal if the safety condition is detected.

[0064] It should be understood that one or more of the aforementioned embodiments of the present invention may be combined as long as the combined embodiments are not mutually exclusive.

[0065] As will be appreciated by those skilled in the art, aspects of the present invention may be implemented as devices, methods, or computer program products. Accordingly, aspects of the present invention may take the form of a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microcode, etc.), or a combination of software and hardware embodiments (all of which may be referred to as "circuits," "modules," or "systems" in general herein). In addition, aspects of the present invention may take the form of a computer program product implemented in one or more computer-readable media having a computer executable code implemented thereon.

[0066] Any combination of one or more computer-readable media may be utilized. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. As used herein, "computer-readable storage medium" encompasses any tangible storage medium that can store instructions that can be executed by a processor or a computing system of a computing device. A computer-readable storage medium may be referred to as a computer-readable non-temporary storage medium. A computer-readable storage medium may also be referred to as a tangible computer-readable medium. In some embodiments, a computer-readable storage medium may also store data that can be accessed by a computing system of a computing device. Examples of computer-readable storage media include, but are not limited to, floppy disks, magnetic hard disk drives, solid-state hard disks, flash memories, USB thumb drives, random access memories (RAM), read-only memories (ROM), optical disks, magneto-optical disks, and register files of computing systems. Examples of optical disks include compact disks (CDs) and digital versatile disks (DVDs), such as CD-ROMs, CD-RWs, CD-Rs, DVD-ROMs, DVD-RWs, or DVD-R disks. The term computer-readable storage medium also refers to various types of recording media that can be accessed by a computer device via a network or communication link. For example, data may be retrieved on a modem, the Internet, or a local area network. Computer executable code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0067] Computer readable signal media may include a propagated data signal with computer executable code implemented therein, for example, in baseband or as part of a carrier wave. These propagated signals may take any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. Computer readable signal media may be any computer readable medium that is not a computer readable storage medium and is capable of conveying, propagating, or transmitting a program used by or in conjunction with an instruction execution system, device, or apparatus.

[0068] "Computer memory" or "memory" is an example of a computer-readable storage medium. Computer memory is any memory that is directly accessible by a computing system. "Computer storage" or "storage" is another example of a computer-readable storage medium. A computer storage is any non-volatile computer-readable storage medium. In some embodiments, a computer storage may also be computer memory, or vice versa.

[0069] As used herein, "computing system" encompasses an electronic component capable of executing a program or machine executable instructions or computer executable code. References to computing systems including examples of "computing system" should be interpreted as possibly including more than one computing system or processing core. A computing system may, for example, be a multi-core processor. A computing system may also refer to a collection of computing systems within a single computer system or distributed among multiple computer systems. The term computing system should also be interpreted as possibly referring to a collection or network of computing devices that each include a processor or computing system. Machine executable code or instructions may be executed by multiple computing systems or processors that may be within the same computing device or may even be distributed across multiple computing devices.

[0070] Machine executable instructions or computer executable code may include instructions or programs that cause a processor or other computing system to perform an aspect of the present invention. Computer executable code for performing operations for multiple aspects of the present invention can be written in any combination of one or more programming languages ​​and compiled into machine executable instructions, and the one or more programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, C++, etc. and conventional procedural programming languages ​​such as "C" programming language or similar programming languages. In some instances, the computer executable code can take the form of a high-level language or take a precompiled form and be used in conjunction with a decoder that generates machine executable instructions at work. In other instances, machine executable instructions or computer executable code can take the form of programming for programmable logic gate arrays.

[0071] The computer executable code may run entirely on the user's computer, partly on the user's computer (as a stand-alone software package), partly on the user's computer and partly on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0072] Aspects of the present invention are described with reference to the flowchart, diagram and / or block diagram of the method, device (system) and computer program product according to an embodiment of the present invention. It should be understood that, when applicable, each frame or a part of the frame of the flowchart, diagram and / or block diagram can be implemented by a computer program instruction in the form of a computer executable code. It should also be understood that, when mutually non-exclusive, the combination of frames in different flowcharts, diagrams and / or block diagrams can be combined. These computer program instructions can be provided to the computing system of a general-purpose computer, a special-purpose computer or other programmable data processing device that produces a machine, so that the instructions run by the computing system of a computer or other programmable data processing device create a mode for implementing the function / action specified in one or more frames of a flowchart and / or block diagram.

[0073] These machine-executable instructions or computer program instructions may also be stored in a computer-readable medium, which can instruct a computer, other programmable data processing apparatus, or other device to work in a specific manner, so that the instructions stored in the computer-readable medium produce a product including instructions for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0074] Machine executable instructions or computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device, thereby producing a computer-implemented process, so that the instructions running on the computer or other programmable apparatus provide a process for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0075] As used herein, a "user interface" is an interface that allows a user or operator to interact with a computer or computer system. A "user interface" may also be referred to as a "human-machine interface device". A user interface may provide information or data to an operator and / or receive information or data from an operator. A user interface may enable input from an operator to be received by a computer and may provide output from the computer to a user. In other words, a user interface may allow an operator to control or manipulate a computer, and the interface may allow a computer to indicate the effects of the operator's control or manipulation. The display of data or information on a display or graphical user interface is an example of providing information to an operator. The reception of data by a keyboard, mouse, trackball, touchpad, pointing stick, graphic input tablet, joystick, game controller, webcam, headset, pedal, wired gloves, remote controller, and accelerometer are all examples of user interface components that implement the reception of information or data from an operator.

[0076] As used herein, "hardware interface" encompasses an interface that enables a computing system of a computer system to interact with and / or control an external computing device and / or apparatus. A hardware interface may allow a computing system to send control signals or instructions to an external computing device and / or apparatus. A hardware interface may also enable a computing system to exchange data with an external computing device and / or apparatus. Examples of hardware interfaces include, but are not limited to, a universal serial bus, an IEEE 1394 port, a parallel port, an IEEE 1284 port, a serial port, an RS-232 port, an IEEE-488 port, a Bluetooth connection, a wireless LAN connection, a TCP / IP connection, an Ethernet connection, a control voltage interface, a MIDI interface, an analog input interface, and a digital input interface.

[0077] As used herein, a "display" or "display device" encompasses an output device or user interface suitable for displaying images or data. A display may output visual, audio, and / or tactile data. Examples of displays include, but are not limited to, computer monitors, television screens, touch screens, tactile electronic displays, Braille screens, cathode ray tubes (CRTs), memory tubes, bi-stable displays, electronic paper, vectorscopes, flat panel displays, vacuum fluorescent displays (VFs), light emitting diode (LED) displays, electroluminescent displays (ELDs), plasma display panels (PDPs), liquid crystal displays (LCDs), organic light emitting diode displays (OLEDs), projectors, and head mounted displays.

[0078] Medical imaging data is defined herein as recorded measurements made by a medical imaging system describing a subject. Medical imaging data can be reconstructed into a medical image. A medical image is defined herein as a reconstructed two-dimensional or three-dimensional visualization of anatomical structure data contained within the medical imaging data. The visualization can be performed using a computer.

[0079] K-space data is defined herein as measurements of radio frequency signals emitted by atomic spins recorded using the antenna of a magnetic resonance apparatus during a magnetic resonance imaging scan. Magnetic resonance data is an example of medical imaging data.

[0080] A Magnetic Resonance Imaging (MRI) image or MR image is defined herein as a reconstructed two-dimensional or three-dimensional visualization of anatomical structure data contained within the magnetic resonance imaging data. The visualization may be performed using a computer. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Preferred embodiments of the present invention will now be described by way of example only and with reference to the accompanying drawings, in which:

[0082] Figure 1 An example of a healthcare system is illustrated;

[0083] Figure 2 The diagram shows the use of Figure 1 A flowchart of a method for a medical system;

[0084] Figure 3 Another example of a medical system is illustrated;

[0085] Figure 4 The diagram shows the use of Figure 4 A flowchart of a method for a medical system;

[0086] Figure 5 illustrates an example of an object sub-image showing a subject lying on a trolley of a magnetic resonance imaging system;

[0087] Figure 6 An example of a search image is shown;

[0088] Figure 7 Draws the Figure 5 An image similar to the image illustrated in In this example, there is a status signal which is a warning indicating a detected risk of pinching fingers of the left hand;

[0089] Figure 8 illustrates the correct detection of a patient posture that does not fit the available examination space; and

[0090] Fig. 9 Detection of a spurious volume that is not part of the patient examination setup is illustrated. DETAILED DESCRIPTION

[0091] Like numbered elements in these figures are equivalent elements or perform the same function. In the case of functional equivalence, elements that have been discussed previously will not necessarily be discussed in later figures.

[0092] Figure 1 An example of a medical system 100 is illustrated. The medical system 100 is shown to include a computer 102. Computer 102 may represent one or more computers located at one or more locations. The medical system 100 is also shown to include a computing system 104. Computing system 104 may also represent one or more computing systems or computing cores disposed in one or more computers located at one or more locations. The computing system 104 is shown to communicate with an optional hardware interface 106 and an optional user interface 108. The hardware interface 106 may enable the computing system 104 to control and communicate with other components (e.g., a medical imaging system). The user interface 108 may enable an operator or control personnel to control and operate the functions of the medical system 100. The computing system 104 is also shown to communicate with a memory 110. The memory 110 may represent different types of memory that may be accessed by the computing system 104. In some examples, the memory 110 is a non-transitory storage medium.

[0093] Memory 110 is shown as containing machine executable instructions 120. Machine executable instructions 120 enable computing system 104 to perform basic data analysis, mathematical, image processing, and control commands. Memory 110 is also shown as containing a first neural network 122, a subject-specific neural network 124, and an object-specific neural network 126. Memory 110 is also shown as containing an overview image 128. Overview image 128 depicts a subject position on a subject support of a medical imaging system. Overview image 128 has a native resolution.

[0094] The first neural network 122 is configured to receive the overview image 128 and process it at a first resolution. For example, the overview image 128 may have its resolution reduced to the first resolution before it is input to the first neural network 122, or the resolution may be automatically reduced using a pooling layer before it is processed by the first neural network 122. In response to receiving the overview image 128 as input, the first neural network 122 outputs a plurality of object locators 130. The object locators may be, for example, identifier locations of a plurality of predetermined objects in the overview image. This may be, for example, locations where bounding boxes or other segmentations are placed to identify the respective objects.

[0095] The memory is also shown as containing an object sub-image 132 created from the overview image 128 using the object locator 130. For example, where the object locator 130 is a bounding box, the object sub-image 132 may be constructed so that it is the same as, encompasses, or includes the bounding box. The object sub-image 132 has a second resolution that is lower than the original resolution and higher than the first resolution. The memory 110 is also shown as containing object position data 134 received from the object-specific neural network 124 in response to receiving the object sub-image 132. The object-specific neural network 124 is programmed to recognize a specific object.

[0096] The memory 110 is also shown as containing a plurality of search images 136 that have been constructed using the object position data 134. The object position data 134 may, for example, contain coordinates of various parts or portions of a specific object. These may be used to identify and define the plurality of search images 136. The memory 110 is also shown as containing locator data 138 received in response to inputting the plurality of search images 136 into the object-specific neural network 126. The object-specific neural network 126 is configured to identify or locate very specific objects, such as a hand bulb, a hand, a string, or other device. The memory is also shown as containing a safety state 140, which is an identification of a specific object identified in one of the plurality of search images 136 using the locator data 138. The safety indicator 140 may be compared to a predetermined criterion 142. If the predetermined criterion 142 is met, a warning signal 144 may be provided. This may be provided to an operator, or may be provided to a computing system in order to adjust the behavior of the medical system.

[0097] Figure 2 The diagram shows the operation Figure 1Flowchart of a method of a medical system 100 of the present invention. First, in step 200, an overview image is received. This may include acquiring it using a camera or an imaging device. Next, in step 202, in response to inputting the overview image 128 into the first neural network 122, an object locator is received. Then, in step 204, an object sub-image 132 is constructed for each of a plurality of predetermined objects using the object locator 130. Next, in step 206, in response to inputting the object sub-image 132 of each of the plurality of predetermined objects, object position data is received from an object-specific neural network. Next, in step 208, a plurality of search images 136 are constructed using the object position data 134. Next, in step 210, in response to inputting the plurality of search images 136, locator data 138 is received from an object-specific neural network. Then, in step 212, a safe state 140 is detected by comparing the locator data and the object position data of each of the predetermined objects relative to a predetermined criterion 142. Finally, in step 214, a warning signal 144 is provided if a safe state is detected.

[0098] Figure 3 Another example of a medical system 300 is shown. Figure 3 The examples in Figure 1 The example in is similar to that in FIG. 3 , except that it additionally includes a magnetic resonance imaging system 302 and an imaging device 326 . The magnetic resonance imaging system 302 is controlled by the computing system 104 .

[0099] The magnetic resonance imaging system 302 includes a magnet 304. The magnet 304 is a superconducting cylindrical magnet having a hole 306 running through it. It is also possible to use different types of magnets; for example, it is also possible to use both a split cylindrical magnet and a so-called open magnet. A split cylindrical magnet is similar to a standard cylindrical magnet, except that the cryostat has been split into two parts to allow access to the isoplane of the magnet, which type of magnet can be used in conjunction with charged particle beam therapy, for example. An open magnet has two magnet parts, one above the other, with a large enough space between the two to receive a subject: the arrangement of the two parts is similar to that of a Helmholtz coil. Open magnets are common because the subject is less constrained. Inside the cryostat of the cylindrical magnet there is a series of superconducting coils.

[0100] There is an imaging region 308 within the bore 306 of the cylindrical magnet 304 within which the magnetic field is strong enough and uniform enough to perform magnetic resonance imaging. A field of view may be located within the imaging region 308. K-space data is typically acquired for the field of view. The region of interest may be the same as the field of view, or it may be a sub-volume of the field of view. A subject 318 is shown supported by a subject support 320 so that at least a portion of the subject 318 may be moved within the imaging region 308.

[0101] There is also a set of magnetic field gradient coils 310 within the bore 306 of the magnet, which is used to acquire preliminary k-space data in order to spatially encode magnetic spins within the imaging region 308 of the magnet 304. The magnetic field gradient coils 310 are connected to a magnetic field gradient coil power supply 312. The magnetic field gradient coils 310 are intended to be representative. Typically, the magnetic field gradient coils 310 include three independent sets of coils for spatial encoding in three orthogonal spatial directions. The magnetic field gradient coil power supply supplies current to the magnetic field gradient coils 310. The current supplied to the magnetic field gradient coils 310 is controlled as a function of time and can be ramped or pulsed.

[0102] Adjacent to the imaging region 308 is a radio frequency coil 314 for manipulating the orientation of magnetic spins within the imaging region 308 and for receiving radio transmissions from spins also within the imaging region 308. The radio frequency antenna may include multiple coil elements. The radio frequency antenna may also be referred to as a channel or antenna. The radio frequency coil 314 is connected to a radio frequency transceiver 316. The radio frequency coil 314 and the radio frequency transceiver 316 may be replaced by separate transmitting coils and receiving coils and separate transmitters and receivers. It should be understood that the radio frequency coil 314 and the radio frequency transceiver 316 are representative. The radio frequency coil 314 is also intended to represent a dedicated transmitting antenna and a dedicated receiving antenna. Similarly, the transceiver 316 may also represent a separate transmitter and receiver. The radio frequency coil 314 may also have multiple receiving elements / transmitting elements, and the radio frequency transceiver 316 may have multiple receiving channels / transmitting channels.

[0103] The imaging device 326 is positioned above the subject 318 and the subject support 320 and is capable of acquiring the overview image 128. The subject support 320 is shown to include a trolley 322 and a main motorized subject support 324. The main motorized subject support 324 is configured to receive the trolley 322 and then insert it into the bore of the magnet 304 with the subject 318. The transceiver 316, the gradient controller 312, the main motorized subject support 324, and the imaging device 326 are shown to be connected to the hardware interface 106 of the computer system 102.

[0104] The memory 110 is also shown as containing pulse sequence commands. Pulse sequence commands are commands or data that can be converted into commands that enable the magnetic resonance imaging system 302 to acquire magnetic resonance data from the imaging region 308. For example, if the subject 318 is inserted into the magnet 304, k-space data can be acquired from the imaging region 308. The memory 110 is also shown as containing measured k-space data 332 acquired by controlling the magnetic resonance imaging system 302 using the pulse sequence commands 330. The memory 110 is also shown as containing magnetic resonance data 334 reconstructed from the measured k-space data 332.

[0105] exist Figure 3 In the embodiment, the magnetic resonance imaging system 302 is used as an example of a medical imaging system. However, a different medical imaging system may be used instead. Figure 3 . For example, the magnetic resonance imaging system may be replaced by any of the following: a computed tomography system, a combined magnetic resonance imaging and computed tomography system, a positron emission tomography system, a combined magnetic resonance imaging and computed tomography system, a single photon emission tomography system, a digital X-ray system, a digital fluoroscopy system, a magnetic resonance imaging-guided radiation therapy system, a computed tomography-guided radiation therapy system, and a magnetic resonance imaging-guided high-intensity focused ultrasound system.

[0106] Figure 4 The diagram shows the use of Figure 3 Flowchart of a method of a medical system 300 of the present invention. In step 400, the imaging device 326 is controlled to acquire the overview image 128. Thereafter, the method proceeds to steps 200, 202, 204, 206, 208, 210, 212 and 214. Additional steps may also be performed thereafter. For example, a subject may be inserted into the bore 306 of the magnetic resonance imaging system 302 using a main automated subject support 324. Thereafter, a pulse sequence command may be used to control the magnetic resonance imaging system to acquire measured k-space data 332. Finally, the measured k-space data may be reconstructed into a magnetic resonance image 334.

[0107] Preparing a radiology (e.g., MR, CT, X-ray) system (medical system 100, 300) and patient (subject 318) for an examination is a time-consuming task that requires a well-trained operator skill. AI-based body and object keypoint detection using a ceiling-mounted camera facing the examination table (imaging device 326) can be used to provide guidance to the operator and automate this task. In this context, the safety protection provided for the examination preparation is very important as it represents a major enabler for automated and autonomous or even remote scanner operation. In particular, moving patients in and out of bores is currently associated with safety risks such as collisions and finger pinching.

[0108] Today, preparation of the patient (subject 318) and movement of the patient into the bore are performed with the operator standing on either side of the patient support. The examination setup is often complex and consists of multiple layers of fixation, comfort, and scanning equipment. Therefore, it is not surprising that many safety incidents have been reported, given the limited field of view of the operator and scheduling time pressures.

[0109] Examples may include one or more of the following features:

[0110] RGB / depth camera (imaging device 326) overview of the exam preparation area, providing a continuous data stream (overview image 128) of the patient exam preparation area (including the patient table and adjacent areas)

[0111] A dedicated set of neural networks (first neural network 122, object-specific neural network 124, and object-specific neural network 126) for detecting body parts and objects in camera images

[0112] A dedicated set of specific system risk areas (as defined in the plurality of search images 136)

[0113] Algorithms for filtering operator-induced signals from relevant patient-induced signals

[0114] Algorithms for detecting unsafe actions or configurations

[0115] An algorithm for classifying the severity of events into at least two categories (e.g., normal and high severity)

[0116] Providing feedback to the operator via the user interface and automatic system response (warning signal 144) to prevent safety incidents

[0117] The example may use a camera sensor (imaging device 326) in conjunction with an algorithm to monitor the examination room in order to detect specific people, objects, and actions associated with safety risks and provide measures to prevent them. Specifically, a series of neural networks are proposed to conduct local investigations of small and micro objects (nurse callers, cable components, connectors, etc.), body parts (e.g., finger joints), and specific system risk areas (table-docking station and table-port joints, cart docking stations, etc.).

[0118] In one example, the overview network (first neural network 122) detects primary patient keypoints (i.e., anatomical landmarks such as joints), which then allows subsequent networks (object-specific neural networks 126) to be run locally (in multiple search images 136) at maximum resolution (between the original resolution and a third resolution) in order to more closely investigate the corresponding body sub-regions.

[0119] In the above, the first neural network 122 provides key points. An alternative to this is that the first neural network 122 provides object locators 130 (e.g., bounding boxes) for a plurality of predetermined objects. These bounding boxes are then used to make object sub-images, and each object is then processed in detail. For example, an object-specific neural network outputs key point data as object position data.

[0120] In an exemplary case, a specific safety risk is associated with a finger grabbing the edge of the examination table during the movement of the examination table into or out of the hole. In particular, locations such as the connection to the motor platform of the examination table support are potentially dangerous locations. In order to further improve safety, it is proposed to use a two-pronged approach, in which the body and fingers are detected immediately, and the dangerous area is also monitored frequently at the highest resolution. This can significantly improve the detection capability because it can enhance the detection when obstacles appear or the patient moves out of the hole after scanning. In another exemplary case, the inspection setting may not be completely in the hole over a longer length. In order to detect this, an overall collision detection algorithm is implemented, which compares the inspection volume and range with the available hole diameter, or generally with the area that is not occupied or used by the system during acquisition. Here, 3D sensing capabilities are used to measure the 3D setting. A key capability of such an algorithm can be to filter non-patient signals from patient signals (i.e., signals belonging to the patient and the setting on the patient table) to avoid false alarms. The following set of algorithms is proposed:

[0121] - Identify areas and contours in the image that do not fit in the available space

[0122] -Determine signal contour location and classify signals as relevant or not relevant to the patient examination setting.

[0123] - Determining a motion correlation of the signal with respect to a relevant system motion (eg, table movement), and using the correlation value to relate the signal to a patient examination setting.

[0124] - Additionally, body keypoint information is used: in case the area does not fit the patient's body the signal is ignored or downgraded.

[0125] The same overview network mentioned earlier also locates related objects and accessories. Safety risks may not only be of mechanical type, but also of RF / thermal type. To reduce RF / thermal safety incidents, close monitoring of the location and routing of RF coil cables is proposed. Such cables must be positioned according to strict manufacturer's instructions to prevent RF safety and thermal injury incidents. However, correct cable routing may not be a simple task, and cable deformation may occur spontaneously and out of the operator's sight. Moreover, remote scanner operation requires monitoring of all cables before and during the movement of the patient in the bore.

[0126] Another feature related to RF safety is the use of a camera to continuously monitor the position of all RF coils. Comparing the detected coils with the coil connection status reported by the system provides a direct way to avoid unconnected coils in the inspection setup.

[0127] Similar accessories, such as the Nurse Call 600 (a device that a patient can use to get the operator's attention) are also monitored ( Figure 1 middle), and the correct position of the hearing protection and communication headset (504) relative to the patient's ears.

[0128] The above detected events may be potentially unsafe, and if detected with a high degree of confidence this information (safety status) is used to:

[0129] - Provide operators with information used to resolve security incidents, including:

[0130] -Spatial location of events

[0131] - Type and context of the event

[0132] - Estimated severity of the risk

[0133] -Suggestions for solutions

[0134] - Overrule automatic system actions for a subset of events of high severity, such as:

[0135] -When the finger is detected to be out of the safe area (close to the clamped position), the automatic inspection platform operation is stopped.

[0136] move

[0137] - Slow down the inspection table when a collision / finger pinching event may occur

[0138] - Allows manual override using only controls close to the patient (e.g., via thumb switch for table movement)

[0139] - Provide patients with information to identify and resolve safety events, including:

[0140] -Automatic voice commands / alerts via speaker system

[0141] -Displays detected areas superimposed on the patient's body to facilitate intuitive positioning by the operator in the real world.

[0142] If there are low detection confidence or undetectable body parts or objects, the camera may not be able to safely and fully intervene in the workflow. For such cases, different system behaviors can be used, for example, when the camera does not detect a hand (e.g. due to detection failure or because it is not visible at all), a general suggestion can be made to the operator to check the respective hand and ensure it is in a safe position, protective gear is applied, etc. Due to the complexity of typical inspection setups (e.g., hands covered by coils or blankets), some of these situations may occur frequently, and blocking the system in such cases will render the system unusable and must be carefully avoided. Therefore, only high confidence detections can be used to reduce the safety incident rate, thereby reducing the unreasonable system blocking rate to a lower user-acceptable level.

[0143] Figure 5 An example of an object sub-image 132 is illustrated, which shows a subject 318 lying on a subject support 320 of a magnetic resonance imaging system. A plurality of anatomical key point locators 500 can be seen. The subject's hands 500 are in a safe position in terms of collision and finger pinching, and the earphones 504 are shown as being correctly placed. However, in this image, the nurse call is not correctly placed in the patient's left hand. Correct patient communication should be established before the subject support and patient are moved into the magnet.

[0144] Figure 6 An example of a search image 136 is shown. In this example, a neural network or object-specific neural network 126 has been trained to detect and locate a nurse call within the reach of a patient's hand. As can be seen, the locator data 138 identifies the location of a nurse call 600 in the subject's hand 502.

[0145] Figure 7 Draws the Figure 5 . In this example, there is a warning signal 700 that outlines the subject's hand as a warning, indicating the risk of fingers being pinched on the left hand. When the hand is close to the connection with the examination table motor platform, the hole and the cart docking station position, the examination table movement is unsafe.

[0146] Figure 8 Illustration of the correct detection of a patient posture that does not fit the available examination space.

[0147] Fig. 9 Detection of a false volume that is not part of the patient examination setup is illustrated. Image 900 has not had the signal from the operator 901 removed. Image 902 illustrates the operator area 904 where the signal from the operator 901 has been identified and removed.

[0148] While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary only and not restrictive; the invention is not limited to the disclosed embodiments.

[0149] By studying the drawings, the specification and the appended claims, those skilled in the art can understand and implement other variations of the disclosed embodiments when practicing the claimed invention. In the claims, the word "comprising" does not exclude other elements or steps, and the word "one" or "an" does not exclude a plurality. A single processor or other unit can implement the functions of several items recorded in the claims. The fact that some measures are recorded in different dependent claims does not mean that a combination of these measures cannot be used to advantage. The computer program may be stored / distributed on a suitable medium, such as an optical storage medium or solid-state medium provided together with or as part of other hardware, but the computer program may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems. Any figure marks in the claims should not be interpreted as limiting the scope.

[0150] Reference numerals list

[0151] 100 Medical System

[0152] 102 Computer

[0153] 104 Computing Systems

[0154] 106 Hardware Interface

[0155] 108 User Interface

[0156] 110 Memory

[0157] 120 machine executable instructions

[0158] 122 The First Neural Network

[0159] 124 Object-Specific Neural Networks

[0160] 126 Object-Specific Neural Networks

[0161] 128 Overview Images

[0162] 130 Object Locator

[0163] 132 object sub-image

[0164] 134 Object location data

[0165] 136 multiple search images

[0166] 138 Locator data

[0167] 140 Safe status

[0168] 142 Predetermined Standards

[0169] 144 Warning Signs

[0170] 200 Receiving an overview image, wherein the overview image has an original resolution and depicts a subject positioned on a subject support of a medical imaging system

[0171] 202 In response to inputting the overview image into the first neural network, receiving an object locator describing the positions and orientations of a plurality of predetermined objects in the overview image

[0172] 204 Using the object locator to construct an object sub-image for each of a plurality of predetermined objects

[0173] 206 In response to inputting the object sub-image of each of the plurality of predetermined objects into an object-specific neural network that is unique to each of the plurality of predetermined objects, receiving object position data describing each of the plurality of predetermined objects

[0174] 208 Constructing multiple search images using object position data of multiple predetermined objects

[0175] 210 In response to inputting each of the plurality of search images into an object-specific neural network, receiving locator data describing the location of the object

[0176] 212 Detecting a safety state by comparing the locator data and the object position data of each of the predetermined objects relative to a predetermined standard

[0177] 214 Provides a warning signal when a safe state is detected

[0178] 300 Medical System

[0179] 302 Magnetic Resonance Imaging System

[0180] 304 Magnet

[0181] 306 magnet hole

[0182] 308 Imaging Area

[0183] 310 Magnetic Field Gradient Coil

[0184] 312 Magnetic Field Gradient Coil Power Supply

[0185] 314 RF Coil

[0186] 316 Transceiver

[0187] 318 subjects

[0188] 320 Subject Support

[0189] 322 Trolley

[0190] 324 Host Automated Subject Support

[0191] 326 Imaging equipment

[0192] 330 Pulse sequence command

[0193] 332 Measured k-space data

[0194] 334 Magnetic resonance images

[0195] 400 Control imaging device to obtain overview image

[0196] 500 anatomical keypoint locators

[0197] 502 Hands

[0198] 504 Headphones

[0199] 600 Nurse Call

[0200] 700 warning signs

[0201] 900 images

[0202] 901 Operator

[0203] 902 images

[0204] 904 Operator Area

Claims

1. A medical system (100, 300), comprising: - a memory (110) storing machine-executable instructions (120), a first neural network (122), a subject-specific neural network (124), and an object-specific neural network (126); - a computing system (104), wherein execution of the machine-executable instructions causes the computing system to: - receiving (200) an overview image (128), wherein the overview image has an original resolution and depicts a subject (318) positioned on a subject support (320) of a medical imaging system (302); - in response to inputting the overview image into the first neural network, receiving (202) an object locator (130) describing the position and orientation of a plurality of predetermined objects in the overview image, wherein the first neural network is configured to process the overview image at a first resolution, wherein the first resolution is lower than the native resolution, wherein the plurality of predetermined objects include the subject and the subject support; - constructing (204) an object sub-image (132) for each of the plurality of predetermined objects using the object locator, wherein each object sub-image has a second resolution, wherein the second resolution is lower than the original resolution and higher than the first resolution; - in response to inputting the object sub-image of each of the plurality of predetermined objects into the object-specific neural network unique to each of the plurality of predetermined objects, receiving (206) object position data (134) describing each of the plurality of predetermined objects, - constructing (208) a plurality of search images (136) using the object position data of the plurality of predetermined objects, wherein the plurality of search images have a third resolution higher than the second resolution, wherein the third resolution is less than or equal to the original resolution; - in response to inputting each of the plurality of search images into the object-specific neural network, receiving (210) locator data (138) describing the location of an object; - detecting (212) a safety state (140) by comparing the locator data and the object position data of each of the predetermined objects relative to a predetermined criterion; and - providing (214) a warning signal (144, 700) in case said safety state is detected.

2. The medical system according to claim 1, wherein: Execution of the machine-executable instructions causes the computing system to display a warning graphical user interface if the warning signal is provided, wherein the warning graphical user interface is configured to perform any of the following: - provide spatial localization of the cause of the safety state, - displaying the warning signal type and / or the warning context using said predetermined criteria, - displaying a severity ranking using said predetermined criteria, - displaying operator instructions retrieved from an instructions database in response to a query using said predetermined criteria, - Provides voice or audible alarms, - projecting a localized warning light onto the subject using a projector, wherein the localized warning light is configured to illuminate an object that triggers the safe state, and - A combination of the above.

3. The medical system according to claim 1 or 2, wherein: A subject-specific neural network for the subject is configured to output a set of anatomical keypoint locators (500), wherein at least a portion of the plurality of search images are determined using a predetermined geometric relationship with the set of anatomical keypoint locators.

4. The medical system according to claim 3, wherein: Execution of the machine-executable instructions further causes the computing system to: - filtering the locator data using a locator filter adapted by the keypoint locator; and - Ignore locator data filtered by the locator filter.

5. A medical system according to any one of the preceding claims, wherein: Any of the following: - the object is a nurse call device (600) squeeze ball; - the object is a headset (504) and / or an earplug, wherein the plurality of search images include a head region of the subject and the locator data describe the position of the object relative to a central locator of the subject's ear. - the object is a conductive cable, wherein the plurality of predetermined objects include a magnetic resonance imaging coil and / or a wired device, wherein the plurality of predetermined objects further include at least one magnetic resonance transmit body coil element, wherein the plurality of search images include an area adjacent to the magnetic resonance imaging coil and / or the wired device, wherein the plurality of search images further include an area adjacent to the at least one transmit body coil element, wherein the plurality of search areas further include additional areas having predetermined positions relative to the set of anatomical keypoint locators; and - the object is a hand, wherein the warning signal is configured for providing a hazardous hand position warning signal if the hand is in a pinched position or a current loop causing position.

6. A medical system according to any one of the preceding claims, wherein: The plurality of predetermined objects include a main motorized patient support (324), wherein the object is an MR cart (322), wherein the plurality of search images are positioned to cover an edge of the main motorized patient support of the MR to check for potential collision with the MR cart when automatically moving the main motorized patient support.

7. A medical system according to any one of the preceding claims, wherein: The object is any of the following: -liner; - Cables; - Breathing tube; - Anatomy; - Intravenous supply; - the object is an MR conditioning device, wherein the positions of the plurality of search images are obtained from a selected magnetic resonance imaging protocol, - a magnetic field sensitive device designed to be kept within a maximum B0 field strength, wherein said search image is located in an area where said B0 field strength is above a predetermined threshold or reaches said magnetic contour; a proximity-sensitive device designed to maintain a minimum distance from a second device, wherein the second device is one of the plurality of predetermined objects, the plurality of search images being located at predetermined positions relative to the object position data of the second device; and - A combination of the above.

8. A medical system according to any one of the preceding claims, wherein: The plurality of predetermined objects include one or more magnetic resonance imaging coils, the localizer data includes radio frequency cable position data, Wherein, execution of the machine executable instructions further causes the computing system to determine the security state by performing any one of the following: - comparing the object position data and the radio frequency position data of the one or more magnetic resonance imaging coils with a set of permissible radio frequency routing paths, - comparing said subject position data of said one or more magnetic resonance imaging coils with a set of reported magnetic resonance imaging coil connections; and - A combination of the above.

9. A medical system according to any one of the preceding claims, wherein: The overview image is any one of: an optical image, a color optical image, and a three-dimensional image.

10. The medical system according to any one of the preceding claims, wherein: The medical system also includes a medical imaging system (302) having the subject support, wherein the medical system also includes an imaging device (326) configured to acquire the overview image by imaging at least the subject support, wherein execution of the machine-executable instructions further causes the computing system to control (400) the imaging device to acquire the overview image.

11. The medical system according to claim 10, wherein: The imaging device is a three-dimensional camera, and execution of the machine-executable instructions further causes the computing system to perform any one of the following: - comparing the spatial profile of the object with a predetermined volume in order to detect a collision condition, - using the contour data from the 3D camera to filter out and ignore localizer data not relevant to the inspection setup, - determining the correlation of the signal motion with respect to the subject support motion in order to filter out and ignore localizer data not relevant to the examination setup, and - A combination of the above.

12. A medical system according to any one of the preceding claims, wherein: The plurality of predetermined objects include an operator (901), and execution of the machine executable instructions further causes the computing system to perform the following: - identifying an operator area in the overview image using the operator's object locator (904); and - Removing and / or ignoring locator data and / or object position data within said operator area.

13. A medical system according to any one of the preceding claims, wherein: The overview image is received as an image stream having a frame rate, the first neural network is configured to process the overview image at a rate equal to or higher than the frame rate, and execution of the machine-executable instructions causes the computing system to repeatedly input the overview image to obtain the object locator at the frame rate.

14. A computer program comprising machine executable instructions (120), a first neural network (122), a subject-specific neural network (124), and an object-specific neural network (126), wherein: Execution of the machine-executable instructions causes the computing system to: - receiving (200) an overview image (128), wherein the overview image has a native resolution and depicts a subject positioned on a subject support of the medical imaging system; - in response to inputting the overview image into the first neural network, receiving (202) an object locator (130) describing the position and orientation of a plurality of predetermined objects in the overview image, wherein the first neural network is configured to process the overview image at a first resolution, wherein the first resolution is lower than the native resolution, wherein the plurality of predetermined objects include the subject and the subject support; - constructing (204) an object sub-image (132) for each of the plurality of predetermined objects using the object locator, wherein each object sub-image has a second resolution, the second resolution being lower than the original resolution and higher than the first resolution; - in response to inputting the object sub-image of each of the plurality of predetermined objects into an object-specific neural network unique to each of the plurality of predetermined objects, receiving (206) object position data (134) describing each of the plurality of predetermined objects, - constructing (208) a plurality of search images (136) using the object position data of the plurality of predetermined objects, wherein the plurality of search images have a third resolution higher than the second resolution, the third resolution being less than or equal to the original resolution; - in response to inputting each of the plurality of search images into an object-specific neural network, receiving (210) locator data (138) describing a location of an object; - detecting (212) a safety state (140) by comparing the locator data and the object position data of each of the predetermined objects relative to a predetermined criterion; and - providing (214) a warning signal (144, 700) in case said safety state is detected.

15. A computer-implemented method, wherein: The method comprises: - receiving (200) an overview image (128), wherein the overview image has a native resolution and depicts a subject positioned on a subject support of the medical imaging system; - in response to inputting the overview image into the first neural network, receiving (202) an object locator (130) describing the position and orientation of a plurality of predetermined objects in the overview image, wherein the first neural network is configured to process the overview image at a first resolution, wherein the first resolution is lower than the native resolution, wherein the plurality of predetermined objects include the subject and the subject support; - constructing (204) an object sub-image (132) for each of the plurality of predetermined objects using the object locator, wherein each object sub-image has a second resolution, the second resolution being lower than the original resolution and higher than the first resolution; - in response to inputting the object sub-image of each of the plurality of predetermined objects into an object-specific neural network unique to each of the plurality of predetermined objects, receiving (206) object position data (134) describing each of the plurality of predetermined objects, - constructing (208) a plurality of search images (136) using the object position data of the plurality of predetermined objects, wherein the plurality of search images have a third resolution higher than the second resolution, the third resolution being less than or equal to the original resolution; - in response to inputting each of the plurality of search images into an object-specific neural network, receiving (210) locator data (138) describing a location of an object; - detecting (212) a safety state (140) by comparing the locator data and the object position data of each of the predetermined objects relative to a predetermined criterion; and - providing (214) a warning signal (144, 700) in case said safety state is detected.

Images