Detecting safety conditions during medical imaging procedures

Abstract

Execution of the machine-executable instructions causes the computing system to receive an overview image, receive an object locator in response to inputting the overview image into a first neural network, construct an object sub-image using the object locator, receive object location data in response to inputting the object sub-image for each of a plurality of predetermined objects into an object identification neural network, construct a plurality of search images using the object location data for the plurality of predetermined objects, receive locator data describing the location of the item in response to inputting each of the plurality of search images into an item identification neural network, and detect a safety condition by comparing the locator data with the object location data for each of the predetermined objects relative to a predetermined criteria.

Description

[Technical Field]

[0001] The present invention relates to medical imaging, and more particularly to safety during medical imaging procedures. [Background technology]

[0002] Various medical imaging techniques, such as magnetic resonance imaging (MRI), computed tomography, positron emission tomography, and single photon emission tomography, allow for detailed visualization of a subject's anatomy. Summary of the Invention [Problem to be solved by the invention]

[0003] In medical imaging modalities such as these, it is important to properly position the subject prior to the procedure, otherwise the subject may be imaged improperly.

[0004] U.S. Patent Application Publication No. 20213 / 012659(A1) discloses a medical device including a medical imaging system configured to acquire medical image data from a subject, a subject support configured to support the subject during acquisition of the medical image data, and an optical imaging system configured to acquire optical image data of the subject on the subject support. Execution of the machine-executable instructions causes a processor controlling the medical device to control the optical imaging system to acquire optical imaging data, generate an initial vector using the optical imaging data, generate a composite image by inputting the initial vector to a generator neural network, calculate a difference between the composite image and the optical imaging data, and provide a warning signal if the difference differs by a predetermined threshold. The generator neural network is trained to generate a composite image of the subject on the subject support in response to input of the initial vector.

[0005] U.S. Patent Application Publication No. 2021 / 0080531 A1 discloses determining scan settings for a localizer scan using a deep learning neural network that involves mapping MR calibration images to corresponding anatomical regions of interest.

[0006] US Patent Application Publication No. 2022 / 0189622 A1 discloses a camera system that uses a camera system and a neural network to identify misplaced objects associated with a medical instrument. [Means for solving the problem]

[0007] The present invention provides a medical system, a computer program and a computer-implemented method according to the independent claims.

[0008] The use of camera systems to monitor the setup and safety of medical imaging systems can pose several challenges. One challenge is that even if the system has a small error rate, the operator is likely to override it. To be useful in a clinical setting, such systems must have near-perfect accuracy. Another difficulty is that properly training the neural networks used to monitor the images can be very labor-intensive.

[0009] The embodiments can have the advantage of being easily tailored to specific clinical situations of medical imaging protocols, as well as having high accuracy. The embodiments can achieve this by first using a hierarchy of neural networks, as illustratively described below, to which an overview image is received. The first neural network first provides object locators to identify a predetermined list of objects in a low-resolution version of the overview image. These object locators are then used to construct object sub-images from the overview image, which are fed to an object-specific neural network. The object sub-images have intermediate resolution. The object locators are then used, possibly with additional information, to construct multiple search images from the overview image. A specialized item-specific neural network is then used to locate specific items or safety conditions within the multiple search images. The use of a hierarchy of neural networks can provide neural networks that can be accurately trained for specific tasks. The hierarchical structure can also allow item-specific neural networks and object-specific neural networks to be easily added or removed from the system.

[0010] In one aspect, the present invention provides a medical system comprising a memory storing machine-executable instructions. The memory further stores a first neural network, an object identification neural network, and an item identification neural network. The medical system further comprises a computing system. Execution of the machine-executable instructions causes the computing system to receive an overview image. The overview image has an original resolution and describes a subject position on a subject support of a medical imaging system. Execution of the machine-executable instructions further causes the computing system to receive, in response to inputting the overview image into the first neural network, object locators describing the origin and orientation of a plurality of predetermined objects in the overview image.

[0011] The first neural network is configured to process the overview image at a first resolution. The first resolution is lower than the original resolution. The plurality of predetermined objects includes a subject and a subject support. Execution of the machine-executable instructions further causes the computing system to construct an object sub-image for each of the plurality of predetermined objects using the object locator. Each object sub-image has a second resolution. The second resolution is lower than the original resolution and higher than the first resolution. Execution of the machine-executable instructions further causes the computing system to receive object location data describing each of the plurality of predetermined objects in response to inputting the object sub-image for each of the plurality of predetermined objects into the object identification neural network.

[0012] The object identification neural network is unique for each of the plurality of predetermined objects. Execution of the machine-executable instructions further causes the computing system to construct a plurality of search images using the object location data of the plurality of predetermined objects. The plurality of search images have a third resolution higher than the second resolution. The third resolution is equal to or less than the original resolution. Execution of the machine-executable instructions further causes the computing system to reserve locator data describing the location of the item in response to inputting each of the plurality of search images to the item identification neural network. Execution of the machine-executable instructions further causes the computing system to detect a safety condition by comparing the locator data and the object location data of each of the predetermined objects against predetermined criteria. Execution of the machine-executable instructions further causes the computing system to provide a warning signal if a safety condition is detected.

[0013] The safe state may be, for example, a flag in software or a label that is set when a predetermined criterion is met. In one specific example, multiple search images can define an area or regions where it is not advantageous for the subject to place their hand. The multiple search images can be searched using an item-specific neural network specifically trained to find hands. If the item-specific neural network detects a hand, it sets a safe state indicating that the particular search image has a hand located within it. The warning signal can take different forms in different embodiments. For example, it can display a warning or additional information in a graphical user interface. In other cases, the warning signal can be used to modify the behavior of the medical system. For example, it can cause the medical system to suspend functions until a command is given by the operator. In other cases, it can disable various functions, such as moving the subject support. The warning signal can also be integrated into a control loop or control system.

[0014] The embodiments can be beneficial because there is a hierarchy of neural networks provided. For example, using the previous example, identifying a hand in a location that may be dangerous to the subject may be difficult to train using a single neural network. The first neural network operates at a lower resolution and can generally identify or label objects, a task that is well known and very well performed by various neural networks. It is also possible to train a neural network with general labels in the image. The next level is an object-specific neural network.

[0015] This may actually be one neural network, or a combination of neural networks working together to look at each object, with a neural network trained specifically for that object. This is then used to provide more information about the direction or object position data. This object position data can be used to define the locations of multiple search images. In some cases, multiple search images may be defined with respect to object position data from one object, or may be from multiple objects. For example, in one example, object position data from two objects may be used to trace a wire traveling between the two objects, or to ensure that the wires have a certain amount of distance between each other. In the case of a subject support, locations on a particular subject support may be used to define multiple search images.

[0016] The different image resolutions for the layers of the neural network can be provided in different ways. For example, there may be an intermediate step where an averaging process is performed to reduce the resolution of the different images. Alternatively, the neural network can do this itself using an average max-pooling layer at the input. That is, for example, the first neural network can have an averaging max-pooling layer that reduces the overview image from its original resolution to the first resolution.

[0017] The overview image can take different forms in different embodiments. In one example, the overview image is a conventional image from a camera or video system. The conventional image can be, for example, an RGB or grayscale image. In this case, progressing between the first resolution, the second resolution, and the third resolution involves converting or reducing the resolution of the overview image when constructing the image to be input to the neural network. As mentioned above, this reduction in resolution can be done when the image is reconstructed, or by pooling layers at the input to one of the neural networks.

[0018] Another possibility is that the overview image includes composite data from multiple image modalities or types of sensors. For example, a camera or video system may provide a three-dimensional image along with an RGB or grayscale image. In this case, the image would have more than two channels of data. It would have three RGB channels and an additional channel with depth data. In this case, the neural network would be configured to accept four channels of data for input instead of three. It is understood that the resolution reduction to the first, second, and third resolutions applies to all channels of the overview image.

[0019] As mentioned above, the object identification neural network can be a single neural network or a chain of neural networks. In another embodiment, multiple search images can be defined by dedicated specific system risk regions. These dedicated specific system risk regions can be defined in terms of relative coordinates of specific objects to the object position data.

[0020] In some specific examples, a warning signal can be used to stop automatic settings of a medical imaging system. For example, it can stop subject support movement when fingers are in an unsafe position or pose. Another example is manipulating a table or subject support when a collision is likely. For example, if there is a possibility of a collision, the table or subject support can be manipulated to have a speed that is less likely to injure someone.

[0021] In a further example, a warning signal can cause a medical system to modify its behavior to only allow manual override by controls close to the patient. For example, there may be a control device used to insert a subject into a medical imaging system. The warning signal can force or modify the system so that only controls adjacent to the patient or subject can move the subject. This would reduce the chances of a more distant operator causing injury to the subject.

[0022] The advantage of using a hierarchy of neural networks is that specialized neural networks are more specific and sensitive to a given capacity. This can have the advantage of increasing the accuracy of the overall system and reducing the occurrence of false alarms. Additionally, neural networks trained individually for individual targets often have the advantage of being complementary, allowing additional objects and objects to be added to the system without changing the detection performance of existing object detection networks. While the first neural network is trained to recognize all objects currently in use, once a particular neural network in the hierarchy has been trained, it does not need to be retrained.

[0023] The neural networks used in this medical system include a first neural network, an object identification neural network, and an item identification neural network, and can be implemented with different architectures. Exemplary architectures of neural networks that can be used generally include elements of ResNet, U-net, and fully convolutional layers. Such fully convolutional neural networks can be particularly beneficial because they have been shown to be particularly well-suited, i.e., have very good sensitivity and specificity, for detecting and locating individual objects against a background of many other objects.

[0024] The first neural network can be trained with images having a first resolution labeled with a plurality of predetermined objects (the labels can include contours, segmentation masks, keypoints, or bounding boxes). The object-specific neural network can be trained with images having a second resolution labeled with location data of the specific objects to which the object-specific neural network is assigned. The item-specific neural network can be trained with images having a third resolution trained with specific items.

[0025] In another embodiment, the use of predetermined criteria can be used to provide a hierarchy of security hazards and risks. For example, the predetermined criteria can provide a ranking of the severity of an event. For example, there can be multiple classes of security hazards, such as normal security hazards and high severity security hazards. The basic response of the medical system can change its function according to these two classes.

[0026] In some examples, a warning signal may be displayed or presented to a user via a visual user interface or other audible system. The warning signal may, for example, be used to control or limit machine use, such as temporarily disabling or disabling a medical system if there is a dangerous situation. Each finding or security risk identified by the system may, for example, be assigned a different type or level of warning. As mentioned above, the warning signal may be a control signal for the medical system.

[0027] In another embodiment, execution of the machine-executable instructions further causes the computing system to display an alert graphical user interface if an alert signal is provided. The alert graphical user interface is configured to perform any one of: providing spatial localization of the cause of the safe condition; displaying an alert signal type and / or alert context using predetermined criteria; displaying a severity ranking using predetermined criteria; displaying operator instructions reviewed from an instruction database in response to a query using predetermined criteria; providing a voice or audible alarm; projecting a local warning light on the subject using a projector; and combinations thereof. Where the local warning light may be configured to illuminate an item that triggers the safe condition, for example, if the subject's hand is in danger of being crushed or pinched, the light may shine on the hand, which may provide an improved means of ensuring that either the subject and / or the operator positions the subject's hand in a safer location.

[0028] In another embodiment, an object-specific neural network for a subject is configured to output a set of anatomical keypoint locators. At least some of the plurality of search images are determined using predetermined geometric relationships or positions relative to the set of anatomical keypoint locators. For example, if testing involves placing a subject's hand in a location that could cause a conductive loop, various anatomical locations where the subject could place their hand in an unfavorable position can be defined relative to the anatomical set of keypoint locators. For example, the positions of the plurality of search images can be defined in terms of the coordinates of the set of anatomical keypoint locators. Once the set of anatomical keypoint locators is determined from the object-specific neural network, they can be used to calculate the positions of the plurality of search images.

[0029] Anatomical keypoints, as used herein, encompass anatomical landmarks or locations located within an object. Often, anatomical keypoint locators are referred to as joint locations. However, anatomical keypoints are broader than joint locations. Anatomical keypoints can incorporate the locations of various joints or anatomical parts of an object. The camera image describes the object and provides a description of the object's exterior or surface. For example, the camera image may be an optical image, an infrared image, or a color image. In another example, the camera image is a three-dimensional surface image.

[0030] In another embodiment, execution of the machine-executable instructions further causes the computing system to filter the locator data using a locator filter modified by the keypoint locator. Execution of the machine-executable instructions further causes the computing system to ignore locator filter data filtered by the locator filter. In this embodiment, the system can be programmed to ignore locator data defined by a particular keypoint locator. This can be very useful, for example, in ignoring data from an operator as opposed to a subject. For example, a subject may be placed on a subject support or table, and an operator may be performing tasks to configure the medical system. Data related to the operator may be ignored or treated separately.

[0031] In another embodiment, the item is a nurse call squeeze valve, and the use of a dedicated neural network to identify this may be beneficial because nurse call squeeze valves are often located in the subject's hand and may otherwise be difficult to locate using general neural networks.

[0032] In another embodiment, the item is a headset and / or earplugs. The plurality of search images includes a head region of the subject. The locator data describes the location of the item relative to a central locator of the subject's ear.

[0033] In another embodiment, the article is a conductive cable. The plurality of predetermined objects includes a magnetic resonance imaging coil and / or a wired device. A wired device in this specification is a device connected to a control or communication wire. The plurality of predetermined objects further includes at least one magnetic resonance transmit body coil element. The plurality of search images includes a region adjacent to the magnetic resonance imaging coil and / or a wired device. The plurality of search images further includes a region adjacent to the at least one transmit body coil element. The plurality of search regions further includes an additional region having a predetermined location relative to a set of anatomical key point locators. This embodiment may be beneficial because it may allow cables to the magnetic resonance imaging coil or wired device to be located away from locations that may cause radio frequency burns to the subject.

[0034] In another embodiment, the item is a hand. The warning signal is configured to provide a dangerous hand position warning signal if the hand is positioned in a pinch position or a current loop culprit position. As used herein, a current loop culprit position encompasses a position where a subject places their hand such that a conductive loop is formed by the subject's body and arm. During magnetic resonance imaging, this can cause radio frequency field-induced currents or locally increased SAR exposure, resulting in heating and / or burns to the subject.

[0035] The search images may be selected to view specific dangerous or high-risk locations. They may identify high-resolution regions using body models and / or other object identification. The search definition region or search images may be defined with respect to whether placing hands on a subject support is dangerous or whether it is dangerous in relation to an anatomical structure such as the waist region.

[0036] In another embodiment, the item is a magnetic resonance conditioned device, such as a device that is allowed for a particular set of magnetic resonance imaging or a particular location for an MRI system. The location for which the magnetic resonance conditioned device is allowed may be derived from a particular cleared use case for a particular magnetic resonance imaging protocol. This may include keeping the device within a B field of a particular strength, searching images to find images that cover up to a particular magnetic isoline, etc. In other cases, this may involve ensuring that a particular distance between different devices is maintained, and the search images may cover problem areas between these devices.

[0037] In another embodiment, the plurality of predetermined objects comprises one or more magnetic resonance imaging coils. The locator data includes radio frequency cable position data. Execution of the machine-executable instructions further causes the computing system to compare the object position data and radio frequency position data of the one or more magnetic resonance imaging coils with a set of allowable radio frequency routing paths, compare the object position data of the one or more magnetic resonance imaging coils with a set of reported magnetic resonance imaging coil connections, and determine a safe state that implements any one of these combinations. The reported magnetic resonance imaging coil connections may be provided, for example, by a radio frequency system of the magnetic resonance imaging system that can detect whether a coil is attached.

[0038] In another embodiment, the overview image is an optical image.

[0039] In another embodiment, the overview image is a color optical image.

[0040] In another embodiment, the overview image is a three-dimensional image, for example it may be acquired using a so-called three-dimensional camera.

[0041] In another embodiment, the medical system further comprises a medical imaging system having a subject support, the medical system further comprises an imaging device configured to acquire an overview image by imaging at least the subject support, and execution of the machine-executable instructions further causes the computing system to control the imaging device to acquire the overview image.

[0042] In another embodiment, the medical imaging system is a magnetic resonance imaging system.

[0043] In another embodiment, the medical imaging system is a computed tomography system.

[0044] In another embodiment, the medical imaging system is a combined magnetic resonance imaging and computed tomography system.

[0045] In another embodiment, the medical imaging system is a positron emission tomography system.

[0046] In another embodiment, the medical imaging system is a combination of a magnetic resonance imaging system and a computed tomography system.

[0047] In another embodiment, the medical imaging system is a single photon emission tomography system.

[0048] In another embodiment, the medical imaging system is a digital x-ray system.

[0049] In another embodiment, the medical imaging system is a digital fluoroscopy system.

[0050] In another embodiment, the medical imaging system is a magnetic resonance imaging guided radiation therapy system.

[0051] In another embodiment, the medical imaging system is a computed tomography guided radiation therapy system.

[0052] In another embodiment, the medical imaging system is a magnetic resonance imaging guided high intensity focused ultrasound system.

[0053] In another embodiment, the medical imaging device is a three-dimensional camera.

[0054] In another embodiment, execution of the machine-executable instructions further causes the computing system to compare the spatial contour of the item to a predetermined volume to detect a collision condition.

[0055] In another embodiment, execution of the machine-executable instructions further causes the computing system to use contour data from a three-dimensional camera to filter out and ignore locator data as relating to a non-inspection setting. For example, the three-dimensional camera can be used to determine or identify objects that are outside the zone of the object support, i.e., not affected by the object transport mechanism (instrument, operator).

[0056] In another embodiment, execution of the machine-executable instructions further causes the computing system to determine signal motion correlation with subject support motion and filter out and disregard locator data as relating to a non-examination setting. For example, when the subject support is moving, portions of the image that are moving can be detected in the three-dimensional image. Areas outside the moving regions can then be identified as relating to a non-examination setting.

[0057] In another embodiment, the plurality of predetermined objects includes an operator. As used herein, an operator encompasses a human operating or controlling a medical system. Execution of the machine-executable instructions further causes the computing system to identify an operator zone within the overview image using the operator's object locator. Execution of the machine-executable instructions further causes the computing system to isolate the locator data and / or object position data within the operator zone. This may be useful, for example, when the subject's anatomical structure is being explored, but it is desirable to ignore the operator's anatomical structure. For example, it may be desirable to ignore the position of the operator's hands when determining whether the subject is at risk of pinching fingers.

[0058] In another embodiment, the overview images are received as an image stream having a frame rate. The first neural network is configured to process the overview images at a rate equal to or greater than the frame rate. Execution of the machine-executable instructions further causes the computing system to repeatedly input the overview images to obtain object locators at the frame rate. In this embodiment, the first neural network operates at a resolution low enough that the images can be processed at a rate equal to or greater than the frame rate. This essentially means that the first neural network can identify objects on a real-time basis.

[0059] In another aspect, the present invention provides a computer program comprising machine-executable instructions, a first neural network, an object identification neural network, and an item identification neural network. Execution of the machine-executable instructions causes a computing system to receive an overview image. The overview image has an original resolution and describes a subject position on a subject support of a medical imaging system. Execution of the machine-executable instructions further causes the computing system to receive, in response to inputting the overview image to the first neural network, object locators describing positions and orientations of a plurality of predetermined objects in the overview image. The first neural network is configured to process the overview image at a first resolution. The first resolution is lower than the original resolution. The plurality of predetermined objects includes a subject and a subject support.

[0060] Execution of the machine-executable instructions further causes the computing system to construct an object sub-image for each of the plurality of predetermined objects using the object locator, each object sub-image having a second resolution, the second resolution being lower than the original resolution and higher than the first resolution. Execution of the machine-executable instructions further causes the computing system to receive descriptive object location data for each of the plurality of predetermined objects in response to inputting the object sub-image for each of the plurality of predetermined objects to an object identification neural network that is unique for each of the plurality of predetermined objects.

[0061] Execution of the machine-executable instructions further causes the computing system to construct a plurality of search images using the object location data of a plurality of predetermined objects. The plurality of search images have a third resolution higher than the second resolution. The third resolution is equal to or less than the original resolution. Execution of the machine-executable instructions further causes the computing system to receive locator data describing the location of the item in response to inputting each of the plurality of search images into the item identification neural network. Execution of the machine-executable instructions further causes the computing system to determine a safety condition by comparing the locator data and the object location data of each of the predetermined objects against predetermined criteria. Execution of the machine-executable instructions further causes the computing system to provide a warning signal if a safety condition is detected.

[0062] In another aspect, the present invention provides a computer-implemented method. The method includes receiving an overview image. The overview image has an original resolution and describes a subject position on a subject support of a medical imaging system. The method further includes receiving, in response to inputting the overview image to a first neural network, object locators describing positions and orientations of a plurality of predetermined objects in the overview image. The first neural network is configured to process the overview image at a first resolution. The first resolution is lower than the original resolution. The plurality of predetermined objects includes the subject and the subject support.

[0063] The method further includes constructing an object sub-image for each of the plurality of predetermined objects using the object locator. Each object sub-image has a second resolution, the second resolution being lower than the original resolution and higher than the first resolution. The method further includes receiving descriptive object location data for each of the plurality of predetermined objects in response to inputting the object sub-image for each of the plurality of predetermined objects into an object identification neural network unique for each of the plurality of predetermined objects. The method further includes constructing a plurality of search images using the object location data for the plurality of predetermined objects.

[0064] The plurality of search images have a third resolution higher than the second resolution. The third resolution is equal to or less than the original resolution. The method further includes receiving locator data describing the location of the items in response to inputting each of the plurality of search images into the item identification neural network. The method further includes detecting a safety condition by comparing the locator data and object position data for each predetermined object against a predetermined criterion. The method further includes providing a warning signal if a safety condition is detected.

[0065] It is understood that one or more of the above-described embodiments of the present invention may be combined, provided that the combined embodiments are not mutually exclusive.

[0066] As will be appreciated by one of skill in the art, aspects of the present invention may be embodied as an apparatus, a method, or a computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be referred to generally herein as a "circuit," "module," or "system." Further, aspects of the present invention may take the form of a computer program product embodied in one or more computer-readable medium(s) having computer-executable code embodied thereon.

[0067] Any combination of one or more computer-readable media may be utilized. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. As used herein, "computer-readable storage medium" encompasses any tangible storage medium capable of storing instructions executable by a processor or computing system of a computing device. The computer-readable storage medium may also be referred to as a computer-readable non-transitory storage medium. The computer-readable storage medium may also be referred to as a tangible computer-readable medium. In some embodiments, the computer-readable storage medium may also be capable of storing data that can be accessed by the computing system of a computing device. Examples of computer-readable storage media include, but are not limited to, floppy disks, magnetic hard disk drives, solid-state hard disks, flash memory, USB thumb drives, random access memory (RAM), read-only memory (ROM), optical disks, magneto-optical disks, and computing system register files. Examples of optical disks include compact discs (CDs) and digital versatile discs (DVDs), such as CD-ROM, CD-RW, CD-R, DVD-ROM, DVD-RW, or DVD-R discs. The term computer-readable storage medium also refers to various types of storage media that can be accessed by a computer device over a network or communications link. For example, data may be retrieved via a modem, over the Internet, or over a local area network. Computer-executable code embodied on a computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wireline, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

[0068] A computer-readable signal medium may include, for example, a propagated data signal having computer-executable code embodied therein, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium is not a computer-readable storage medium but may be any computer-readable medium that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0069] "Computer memory" or "memory" is an example of a computer-readable storage medium. Computer memory is any memory directly accessible to a computing system. "Computer storage" or "storage" is a further example of a computer-readable storage medium. Computer storage is any non-volatile memory computer-readable storage medium. In some embodiments, computer storage may be computer memory, and vice versa.

[0070] As used herein, a "computing system" encompasses electronic components capable of executing programs, machine-executable instructions, or computer-executable code. References to a computing system, including examples of a "computing system," should be interpreted as including two or more computing systems or processing cores, as the case may be. A computing system may be, for example, a multi-core processor. A computing system may also refer to a collection of computing systems within a single computer system or distributed among multiple computer systems. The term computing system should also be interpreted as referring to a collection or network of computing devices, possibly each comprising a processor or computing system. Machine-executable code or instructions may be executed by multiple computing systems or processors, which may be within the same computing device or distributed across multiple computing devices.

[0071] Machine-executable instructions or computer-executable code may comprise instructions or programs that cause a processor or other computing system to perform aspects of the present invention. Computer-executable code for performing operations for aspects of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages, and compiled into machine-executable instructions. In some cases, the computer-executable code may be in the form of a high-level language or in pre-compiled form and may be used in conjunction with an interpreter that generates machine-executable instructions on the fly. In other cases, the machine-executable instructions or computer-executable code may form a program for a programmable logic gate array.

[0072] The computer executable code may run entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider).

[0073] Aspects of the present invention will be described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block or portion of a block in the flowcharts, diagrams, and / or block diagrams, where applicable, can be implemented by computer program instructions in the form of computer-executable code. Furthermore, it should be noted that combinations of blocks in different flowcharts, diagrams, and / or block diagrams can be combined, if not mutually exclusive. These computer program instructions can be provided to a general-purpose computer, special-purpose computer, or other programmable data processing device computing system to produce a machine, such that the instructions, executed via the computer or other programmable data processing device computing system, create means for performing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0074] These machine-executable instructions or computer program instructions may be stored on a computer-readable medium that can direct a computer, other programmable data processing apparatus, or other device to function in a particular manner, such that the instructions stored on the computer-readable medium produce an article of manufacture including instructions that implement the function / acts specified in a block or blocks of the flowcharts and / or block diagrams.

[0075] The machine-executable instructions or computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps executed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, such that the instructions executing on the computer or other programmable apparatus provide a process for implementing the functions / operations specified in the flowcharts and / or block diagrams or blocks.

[0076] As used herein, a "user interface" is an interface that allows a user or operator to interact with a computer or computer system. A "user interface" is sometimes referred to as a "human interface device," and a user interface can provide information or data to an operator and / or receive information or data from an operator. A user interface can allow input from an operator to be received by a computer and can provide output from the computer to a user. In other words, a user interface can allow an operator to control or manipulate a computer, and an interface can allow a computer to show the effects of the operator's control or manipulation. The display of data or information on a display or graphical user interface is an example of providing information to an operator. Receiving data via a keyboard, mouse, trackball, touchpad, pointing stick, graphics tablet, joystick, gamepad, webcam, headset, pedals, wired gloves, remote control, and accelerometer are all examples of user interface components that allow information or data to be received from an operator.

[0077] As used herein, a "hardware interface" encompasses an interface that allows a computer system to interact with and / or control external computing devices and / or equipment. A hardware interface may allow a computer system to send control signals or instructions to external computing devices and / or equipment. A hardware interface may also allow a computing system to exchange data with external computing devices and / or equipment. Examples of hardware interfaces include, but are not limited to, a universal serial bus, an IEEE 1394 port, a parallel port, an IEEE 1284 port, a serial port, an RS-232 port, an IEEE-488 port, a Bluetooth connection, a wireless local area network connection, a TCP / IP connection, an Ethernet connection, a control voltage interface, a MIDI interface, an analog input interface, and a digital input interface.

[0078] As used herein, "display" or "display device" encompasses an output device or user interface adapted to display images or data. A display can output visual, audio, and / or tactile data.

[0079] Examples of displays include, but are not limited to, computer monitors, television screens, touch screens, tactile electronic displays, Braille screens, cathode ray tubes (liquid), memory tubes, bi-stable displays, electronic paper, vector displays, flat panel displays, vacuum fluorescent displays (VF), light emitting diode (LED) displays, electroluminescent displays (ELD), plasma display panels (PDP), liquid crystal displays (LCD), organic light emitting diode displays (OLED), projectors, and head mounted displays.

[0080] Medical imaging data is defined herein as recorded measurements made by a medical imaging system that describe a subject. Medical image data may be reconstructed into a medical image. A medical image is defined herein as a reconstructed two-dimensional or three-dimensional visualization of anatomical data contained within the medical imaging data. This visualization may be performed using a computer.

[0081] K-space data is defined herein as the recorded measurements of radio frequency signals emitted by atomic spins using the antenna of a magnetic resonance machine during a magnetic resonance imaging scan. Magnetic resonance data is an example of medical imaging data.

[0082] A magnetic resonance imaging (MRI) image or MR image is defined herein as a reconstructed two-dimensional or three-dimensional visualization of anatomical data contained within magnetic resonance imaging data, which visualization can be performed using a computer.

[0083] In the following, preferred embodiments of the invention will be described, by way of example only, with reference to the drawings, in which: [Brief explanation of the drawings]

[0084] [Figure 1] 1 shows an example of a medical system. [Figure 2] 2 shows a flowchart illustrating a method of using the medical system of FIG. 1. [Figure 3] 1 illustrates a further example of a medical system. [Figure 4] 5 shows a flowchart illustrating a method of using the medical system of FIG. [Figure 5] 1 shows an example of an object sub-image showing an object lying on a carriage for a magnetic resonance imaging system. [Figure 6] 1 shows an example of a search image. [Figure 7] An image similar to that shown in Figure 5 is shown. In this example, there is a status signal that is a warning indicating a pinch risk detected for the left hand. [Figure 8] We demonstrate accurate detection of patient postures that are incompatible with the available examination space. [Figure 9] 1 illustrates the detection of spurious volumes that are not part of the patient exam setup. DETAILED DESCRIPTION OF THE INVENTION

[0085] Like numbered elements in these figures are equivalent elements or perform the same function. An element as described above is not necessarily discussed in a subsequent figure if there is functional equivalence.

[0086] FIG. 1 illustrates an example medical system 100. The medical system 100 is shown to include a computer 102. The computer 102 may represent one or more computers in one or more locations. The medical system 100 is further shown to include a computing system 104. The computing system 104 may in turn represent one or more computing systems or computing cores located in one or more computers in one or more locations. The computing system 104 is shown to be in communication with an optional hardware interface 106 and an optional user interface 108. The hardware interface 106 may enable the computing system 104 to control and communicate with other components, such as a medical imaging system. The user interface 108 may allow an operator or controller to control and operate functions of the medical system 100. The computing system 104 is further shown to be in communication with a memory 110. The memory 110 may represent different types of memory that the computing system 104 may have access to. In some examples, the memory 110 is a non-transitory storage medium.

[0087] The memory 110 is shown as including machine-executable instructions 120. The machine-executable instructions 120 enable the computing system 104 to perform basic data analysis, mathematics, image processing, and control commands. The memory 110 is further shown as including a first neural network 122, an object identification neural network 124, and an item identification neural network 126. The memory 110 is further shown as including an overview image 128. The overview image 128 is indicative of a subject position on a subject support of a medical imaging system. The overview image 128 has original resolution.

[0088] The first neural network 122 is configured to receive the overview image 128 and process it at a first resolution. For example, the overview image 128 can have its resolution reduced to the first resolution before being input to the first neural network 122, or a pooling layer can be used to automatically reduce the resolution before being processed by the first neural network 122. In response to receiving the overview image 128 as input, the first neural network 122 outputs a plurality of object locators 130. The object locators may be, for example, identifier positions of a plurality of predetermined objects within the overview image. This may be, for example, an arrangement of bounding boxes or other segmentations to identify the locations of various objects.

[0089] The memory 110 is further shown as including object sub-images 132 created from the overview image 128 using the object locator 130. For example, if the object locator 130 is a bounding box, the object sub-images 132 may be configured such that they are identical to, encompass, or include the bounding box. The object sub-images 132 have a second resolution that is lower than the original resolution and higher than the first resolution. The memory 110 is further shown as including object location data 134 received from the object identification neural network 124 in response to receiving the object sub-images 132. The object identification neural network 124 is programmed to identify specific objects.

[0090] The memory 110 is further shown to include a plurality of search images 136 constructed using the object location data 134. The object location data 134 may include, for example, coordinates of various parts or portions of a particular object, which may be used to identify and define the plurality of search images 136. The memory 110 is further shown to include locator data 138 received in response to inputting the plurality of search images 136 into the item identification neural network 126. The item identification neural network 126 is configured to identify or locate highly specific items, such as hand valves, hands, wires, or other devices. The memory is further shown to include a safety status 140, which is the identification of a specific item identified in one of the plurality of search images 136 using the locator data 138. The safety indicator 140 may be compared to predetermined criteria 142. If the predetermined criteria 142 are met, a warning signal 144 may be provided, which may be provided to an operator or to a computing system to modify the behavior of the medical system.

[0091] FIG. 2 shows a flowchart illustrating a method of operating the medical system 100 of FIG. 1. First, in step 200, an overview image is received. This may include capturing it using a camera or imaging device. Next, in step 202, an object locator is received in response to inputting the overview image 128 into a first neural network 122. Next, in step 204, an object sub-image 132 is constructed for each of a plurality of predetermined objects using the object locator 130. Next, in step 206, object location data is received from an object identification neural network in response to inputting the object sub-image 132 for each of a plurality of predetermined objects. Next, in step 208, a plurality of search images 136 are constructed using the object location data 134. Next, in step 210, locator data 138 is received from the item identification neural network in response to inputting the plurality of search images 136. Next, in step 212, a safe condition 140 is detected by comparing the locator data and the object position data for each of the given objects against predetermined criteria 142. Finally, in step 214, if a safe condition is detected, a warning signal 144 is provided.

[0092] Figure 3 shows a further example of a medical system 300. The example of Figure 3 is similar to the example shown in Figure 1, except that it further includes a magnetic resonance imaging system 302 and an imaging device 326. The magnetic resonance imaging system 302 is controlled by the computing system 104.

[0093] The magnetic resonance imaging system 302 includes a magnet 304. The magnet 304 is a superconducting cylindrical magnet with a bore 306. Different types of magnets can be used, including both split cylindrical magnets and so-called open magnets. Split cylindrical magnets are similar to standard cylindrical magnets except that the cryostat is divided into two sections to allow access to the magnet's equal surface. Such magnets can be used, for example, in conjunction with charged particle beam therapy. Open magnets have two magnet sections, one above the other, with a space between them large enough to accommodate the subject, i.e., an arrangement of the two sections similar to the area of ​​a Helmholtz coil. Open magnets are popular because they provide less subject confinement. Inside the cryostat of a cylindrical magnet is a collection of superconducting coils.

[0094] Within the bore 306 of the cylindrical magnet 304 is an imaging zone 308 where the magnetic field is strong and sufficiently uniform to perform magnetic resonance imaging. A field of view may be within the imaging zone 308. K-space data is typically acquired for the field of view. The region of interest may be the same as the field of view or may be a subvolume of the field of view. A subject 318 is shown supported by a subject support 320 to allow movement of at least a portion of the subject 318 within the imaging zone 308.

[0095] Also within the magnet bore 306 are a set of magnetic field gradient coils 310 used for preliminary k-space data acquisition for spatially encoding magnetic spins within the imaging zone 308 of the magnet 304. The gradient coils 310 are connected to a gradient coil power supply 312. The magnetic field gradient coils 310 are intended to be representative. Typically, the magnetic field gradient coils 310 include three separate coil sets for spatially encoding in three orthogonal spatial directions. The gradient power supply supplies current to the gradient coils. The current supplied to the magnetic field gradient coils 310 is controlled as a function of time and can be ramped or pulsed.

[0096] Adjacent to the imaging zone 308 is a radio frequency coil 314 for manipulating the direction of magnetic spins within the imaging zone 308 and for receiving radio transmissions from the spins within the imaging zone 308. A radio frequency antenna may include multiple coil elements. A radio frequency antenna may also be referred to as a channel or antenna. The radio frequency coil 314 is connected to a radio frequency transceiver 316. The radio frequency coil 314 and the radio frequency transceiver 316 may be replaced by separate transmit and receive coils and separate transmitters and receivers. It is understood that the radio frequency coil 314 and the radio frequency transceiver 316 are representative. The radio frequency coil 314 is also intended to represent a dedicated transmit antenna and a dedicated receive antenna. Similarly, the transceiver 316 may also represent a separate transmitter and receiver. The radio frequency coil 314 may also have multiple receive / transmit elements, and the radio frequency transceiver 316 may have multiple receive / transmit channels.

[0097] An imaging device 326 is positioned above the subject 318 and subject support 320 and is capable of acquiring an overview image 128. The subject support 320 is shown as including a table trolley 322 and a main motorized subject support 324. The main motorized subject support 324 is configured to receive the table trolley 322 and then insert it, along with the subject 318, into the bore of the magnet 304. The transceiver 316, tilt controller 312, main motorized subject support 324, and imaging device 326 are shown connected to the hardware interface 106 of the computer system 102.

[0098] Memory 110 is further shown as containing pulse sequence commands. Pulse sequence commands are commands or data that can be converted into commands that enable magnetic resonance imaging system 302 to acquire magnetic resonance data from imaging zone 308. For example, when subject 318 is inserted into magnet 304, k-space data can be acquired from imaging zone 308. Memory 110 is further shown as containing measured k-space data 332 acquired by controlling magnetic resonance imaging system 302 with pulse sequence commands 330. Memory 110 is further shown as containing magnetic resonance data 334 reconstructed from measured k-space data 332.

[0099] In Figure 3, magnetic resonance imaging system 302 is used as an example of a medical imaging system. However, different medical imaging systems can be substituted for the magnetic resonance imaging system shown in Figure 3. For example, the magnetic resonance imaging system can be replaced with any one of a computed tomography system, a combined magnetic resonance imaging and computed tomography system, a positron emission tomography system, a combined magnetic resonance imaging and computed tomography system, a single photon emission tomography system, a digital x-ray system, a digital fluoroscopy system, a magnetic resonance imaging-guided radiotherapy system, a computed tomography-guided radiotherapy system, and a magnetic resonance imaging-guided high intensity focused ultrasound system.

[0100] 4 shows a flowchart illustrating a method of using the medical system 300 of FIG. 3. In step 400, the imaging device 326 is controlled to acquire the overview image 128. After this, the method proceeds to steps 200, 202, 204, 206, 208, 210, 212, and 214. Additional steps may be performed after this. For example, the main motorized subject support 324 may be used to insert the subject into the bore 306 of the magnetic resonance imaging system 302. Pulse sequence commands may then be used to control the magnetic resonance imaging system to acquire measured k-space data 332. Finally, the measured k-space data may be reconstructed into a magnetic resonance image 334.

[0101] Preparing a radiology (e.g., MR, CT, X-ray) system (medical system 100, 300) and patient (subject 318) for an examination is a time-consuming task requiring the trained skills of an operator. AI-based body and object keypoint detection using a ceiling-mounted, table-facing camera (imaging device 326) can be used to provide guidance to the operator and automate this task. Safeguarding examination preparation is of strong interest in this context, as it represents a key enabler for automation and autonomous or remote scanner operation. In particular, moving a patient in and out of the bore currently involves safety risks such as collisions and finger entrapment.

[0102] Today, both patient preparation and movement into the bore are performed while standing on either side of the patient support. Exam setups are typically complex, consisting of multiple layers of immobilization, comfort, and scanning devices. Therefore, given the operator's limited view and scheduling time pressures, it is not surprising that many safety incidents are reported.

[0103] Examples include: an RGB / depth camera (imager 326) overviewing the examination preparation area, providing a continuous data stream (overview image 128) of the patient examination preparation area, including the patient table and adjacent areas; a dedicated set of neural networks (first neural network 122, object specific neural network 124, and item specific neural network 126) for body part and object detection in camera images; a dedicated set of specific system risk regions (defined by a plurality of search images 136); an algorithm for filtering operator-attributed signals from associated patient-attributed signals; algorithms for detecting dangerous behavior or settings; an algorithm for classifying the severity of an event into at least two classes (e.g., normal and high severity); and Preventing safety incidents by providing feedback via the user interface to the operator and automatic system reactions (warning signals 144) The present invention may comprise one or more of the following features:

[0104] In an example, a camera sensor (image capture device 326) can be used to monitor an examination room in combination with algorithms that detect specific people, objects, and actions associated with safety risks and provide preventative action. Specifically, the use of cascades of neural networks for localized surveillance of small and delicate objects (e.g., nurse call buttons, cable components, connectors, body parts, knuckles) and specific system risk areas (e.g., table dock and bore joints, trolley docks, etc.) is proposed.

[0105] In one example, the overview network (first neural network 122) detects key patient keypoints (i.e., anatomical landmarks such as joints), which then allows a subsequent network (item-specific neural network 126) to run locally (on multiple search images 136) at full resolution (between the original resolution and the third resolution) for closer investigation of each body-specific region.

[0106] In the above, the first neural network 122 provides keypoints. An alternative to this is for the first neural network 122 to provide object locators 130 (e.g., bounding boxes) for a given number of objects. These bounding boxes are then used to create object sub-images for each object for further processing. For example, the object identification neural network outputs keypoint data as object location data.

[0107] In one exemplary case, a particular safety risk is associated with fingers gripping the edge of the table while it moves in and out of the bore. Specifically, potentially dangerous locations include the junction of the table to the motor stage supporting the table. Following body and finger detection, a two-prong approach is proposed to further improve safety. This can significantly improve detection capabilities, as it enhances detection in the case of occlusions or when the patient is moving out of the bore after scanning. In another exemplary case, the examination setup may not fit completely within the bore on a larger scale. To detect areas that are not occupied or used by the system during acquisition, a visual collision detection algorithm is implemented that compares the available bore diameter, or generally the examination volume and area. Here, 3D sensing capabilities are used to measure the setup in 3D. A key capability of this algorithm may be its ability to filter out non-patient signals from patient signals (i.e., signals belonging to the patient and the setup on the patient table) to avoid false positives. determining areas and contours of the image that do not fit in the available space; determining signal contour locations and classifying the signals as associated with a patient examination setting; determining a correlation of the signal movement to relevant system movements, e.g., table movement, and using the correlation value to determine whether to associate the signal with a patient examination setting; and Using body keypoint information, i.e., if the region does not fit the patient's body, the signal is ignored or downgraded. A set of algorithms is proposed.

[0108] The same general network described above also localizes related objects and accessories. Safety risks may be mechanical as well as RF / thermal types. To reduce RF / thermal safety events, it is suggested that the location and routing of RF coil cables be closely monitored. Such cables must be placed according to strict manufacturer instructions to ensure RF safety and prevent burn accidents. However, correct cable routing may not be a simple task, and cable deformations may occur spontaneously and become invisible to the operator. Remote scanner operation also requires monitoring of all cables before and during patient movement within the bore.

[0109] Another feature of RF safe connection is the constant monitoring of all RF coil positions using a camera. Comparison of detected coils with the system-reported coil connection status provides an immediate means to avoid unconnected coils in the examination setup.

[0110] Similar accessories such as a nurse call 600 (a device that a patient can use to elicit attention from an operator) are also monitored (center of Figure 1), as well as the proper placement of hearing protection and communication headsets (504) on the patient's ears.

[0111] The above detected events are potentially unsafe and this information (safe state) is provided when detected with high confidence. spatial localization of events, The type and context of the event, the estimated severity of the risk, and Solution providing operators with information to resolve safety events, including If it detects that a finger is hitting something outside the safe area, the automatic table movement stops near the pinch position. Slowing down the table when collision / finger pinch events are likely to occur, Allowing manual override only on controls close to the patient (e.g., via thumb switches for table movement), Providing patients with information to identify the resolution of safety events; Automatic voice instructions / warnings via speaker system, and Detected regions are overlaid on the patient's body to facilitate intuitive localization by the operator in the real world Automated system behavior for a subset of high severity events such as Refuse.

[0112] In the presence of low-confidence detection or undetectable body parts or objects, the camera may not be able to safely and appropriately intervene in the workflow. In such situations, for example, if a hand is not detected by the camera (e.g., because detection failed or simply because it is not visible at all), a different system behavior can be used, with a general recommendation to the operator to check each hand and ensure it is in a safe position, protective devices are applied, etc. Some of these situations may occur regularly due to the complexity of a typical inspection setup (e.g., hands covered by coils or blankets), and blocking the system in such situations would render it unusable and therefore must be carefully avoided. Therefore, using only high-confidence detection, the safe incidence angle can be reduced, thereby reducing the rate of unwarranted system blockage to a low, user-acceptable level.

[0113] 5 shows an example of a subject sub-image 132 showing a subject 318 lying on a subject support 320 for a magnetic resonance imaging system. Numerous anatomical keypoint locators 500 are visible. The subject's hand 500 is in a safe position with respect to collisions and pinching, and the headset 504 is shown to be correctly positioned. However, in this image, the nurse call button is not properly seated on the patient's left hand. Proper patient communication should be established before moving the subject support and patient into the magnet.

[0114] 6 shows an example of a search image 136, in this example a neural network or item specific neural network 126 trained to detect and locate a nurse call button within reach of a patient. It can be seen that locator data 138 identifies the location of a nurse call button 600 in a subject's hand 502.

[0115] Figure 7 shows an image similar to that shown in Figure 5. In this example, there is a warning signal 700 outlining the subject's hand as a warning with the fingers pointing out the risk of pinching the left hand. As the hand approaches the joints to the table motor stage, bore, and trolley dock locations, steady movement is unsafe.

[0116] FIG. 8 shows the accurate detection of patient postures that do not fit into the available examination space.

[0117] 9 illustrates the detection of spurious volumes that are not part of the patient examination setup. Image 900 shows no signal removed from operator 901. Image 902 shows operator zone 904 where signal from operator 901 has been identified and removed.

[0118] While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive, and the invention is not limited to the disclosed embodiments.

[0119] Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. A computer program can be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state medium, together with or supplied as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. Any reference signs in the claims should not be construed as limiting the scope. [Explanation of symbols]

[0120] 100 Medical Systems 102 Computer 104 Computing Systems 106 Hardware Interface 108 User Interface 110 memory 120 machine-executable instructions 122 The First Neural Network 124 Object Identification Neural Network 126 Item-specific neural network 128 overview images 130 Object Locator 132 object sub-images 134 Object location data 136 Multiple Search Images 138 Locator Data 140 Safe State 142 Prescribed standards 144 Warning Signals 200 receiving an overview image, said overview image having original resolution and describing a subject positioned on a subject support of a medical imaging system; 202 receiving object locators describing the positions and orientations of a plurality of predetermined objects within the overview image in response to inputting the overview image into the first neural network. 204. Using the object locator to construct an object sub-image for each of a plurality of predetermined objects. 206. receiving descriptive object location data for each of the plurality of predetermined objects in response to inputting the object sub-image for each of the plurality of predetermined objects into an object specific neural network unique for each of the plurality of predetermined objects. 208 constructing a plurality of search images using object position data for a plurality of predetermined objects; 210 receiving locator data describing the location of the item in response to inputting each of the plurality of search images into the item identification neural network. 212 detecting a safe condition by comparing the locator data and the object position data for each of the predetermined objects against predetermined criteria. 214 Steps for providing a warning signal when a safe condition is detected 300 Medical Systems 302 Magnetic Resonance Imaging Device 304 Magnet 306 Magnet Bore 308 Imaging Zone 310 Magnetic Gradient Coil 312 Gradient magnetic field coil power supply 314 High Frequency Coil 316 Transceiver 318 Subject 320 Subject Support 322 Table Trolley 324 Main motorized subject support device 326 Imaging Device 330 Pulse Sequence Commands 332 Measured k-space data 334 Magnetic Resonance Imaging (MRI) 400 Control the imaging device to acquire an overview image 500 Anatomical Keypoint Locators 502 hands 504 Headset 600 Nurse Call 700 Warning Signal 900 images 901 Operator 902 images 904 Operator Zone

Claims

1. 1. A medical system comprising: a memory storing machine-executable instructions, a first neural network, an object identification neural network, and an item identification neural network; a computing system, wherein execution of the machine-executable instructions causes the computing system to: receiving an overview image, said overview image having original resolution and describing a subject positioned on a subject support of a medical imaging system; receiving object locators describing positions and orientations of a plurality of predetermined objects in the overview image in response to inputting the overview image to the first neural network, the first neural network being configured to process the overview image at a first resolution, the first resolution being lower than the original resolution, the plurality of predetermined objects comprising the subject and the subject support; constructing an object sub-image for each of the plurality of predetermined objects using the object locator, each object sub-image having a second resolution that is lower than the original resolution and higher than the first resolution; receiving object position data describing each of the plurality of predetermined objects in response to inputting the object sub-image for each of the plurality of predetermined objects into the object identification neural network specific to each of the plurality of predetermined objects; constructing a plurality of search images for the plurality of predetermined objects using the object location data, the plurality of search images having a third resolution higher than the second resolution, the third resolution being lower than or equal to the original resolution; receiving locator data describing the location of an item in response to inputting each of the plurality of search images into the item identification neural network; detecting a safe condition by comparing object position data and the locator data for each of the predetermined objects against predetermined criteria; providing a warning signal if the safe condition is detected; a computing system for executing the A medical system having:

2. When the warning signal is provided, execution of the machine-executable instructions causes the computing system to display a warning graphical user interface, the warning graphical user interface comprising: providing a spatial localization of the cause of the safety condition; displaying a warning signal type and / or a warning context using said predetermined criteria; displaying a severity ranking using said predetermined criteria; displaying operator instructions retrieved from an instruction database in response to a query using said predetermined criteria; providing a voice or audible alert; projecting a localized warning light onto the subject using a projector, the localized warning light configured to illuminate an item that triggered the safety condition; A combination of these The medical system of claim 1 , configured to perform any one of the following:

3. 3. The medical system of claim 1, wherein the object-specific neural network for the subject is configured to output a set of anatomical keypoint locators, and wherein at least some of the search images are determined using a predetermined geometric relationship to the set of anatomical keypoint locators.

4. Execution of the machine-executable instructions causes the computing system to: filtering the locator data using a locator filter modified by the keypoint locator; ignoring locator data filtered by the locator filter; The medical system according to claim 3, wherein the medical system executes the above.

5. the item being a nurse call squeeze valve; the items are headsets and / or earplugs, the plurality of search images include a head region of the subject, and the locator data describes a position of the items relative to a central locator of the subject's ear; the item is a conductive cable, the plurality of predetermined objects comprise a magnetic resonance imaging coil and / or a wired device, the plurality of predetermined objects further comprise at least one magnetic resonance transmit body coil element, the plurality of search images comprise regions adjacent to the magnetic resonance imaging coil and / or the wired device, the plurality of search images further comprise regions adjacent to the at least one transmit body coil element, and the plurality of search regions further comprise further regions having predetermined positions relative to the set of anatomical key point locators; the item is a hand, and the warning signal is configured to provide a dangerous hand position warning signal when the hand is positioned in a pinch position or a current loop causing position.

5. The medical system according to claim 1, wherein the medical system is one of:

6. 6. The medical system of claim 1, wherein the plurality of predetermined objects comprises a main motorized patient support, the item is an MR table trolley, and the plurality of search images are positioned to cover a rim of the main motorized patient support of the MR to check for potential collisions with the MR table trolley when the main motorized patient support is automatically moved.

7. The item is: Cushion and Cable and A breathing tube, Anatomical structure and Intravenous supplies; an MR-conditional device, wherein the locations of the plurality of search images are derived from a selected magnetic resonance imaging protocol; a magnetic field sensing device designed to stay within a maximum B field strength, the search image being located in a region where the B field strength is above a predetermined threshold or up to a magnetic isoline; a proximity sensing device designed to maintain a minimum distance from a second device, the second device being one of the plurality of predetermined objects, and the plurality of search images being positioned at predetermined locations relative to the object position data of the second device; A combination of these 7. The medical system according to claim 1, wherein the medical system is one of:

8. the plurality of predetermined objects comprise one or more magnetic resonance imaging coils, and the locator data comprises radio frequency cable position data; Execution of the machine-executable instructions causes the computing system to: comparing the object position data and the radio frequency position data of the one or more magnetic resonance imaging coils to a set of allowable radio frequency routing paths; comparing the object position data of the one or more magnetic resonance imaging coils with a set of reported magnetic resonance imaging coil connections; A combination of these determining the safe state by performing any one of the following: The medical system according to any one of claims 1 to 7.

9. The medical system according to claim 1 , wherein the overview image is any one of an optical image, a color optical image, and a three-dimensional image.

10. 10. The medical system of claim 1, further comprising a medical imaging system having the subject support, the medical system further comprising an imaging device configured to acquire the overview image by imaging at least the subject support, and execution of the machine-executable instructions causes the computer system to control the imaging device to acquire the overview image.

11. the imaging device is a three-dimensional camera, and execution of the machine-executable instructions further comprises: comparing the spatial contour of the item with a predetermined volume to detect a collision condition; using contour data from the 3D camera to filter out and ignore locator data as relating to a non-inspection setting; determining signal motion correlation with subject support motion to filter out and ignore locator data as relating to a non-examination setting; A combination of these The medical system according to claim 10, wherein the medical system executes any one of the following:

12. The plurality of predetermined objects comprises an operator, and execution of the machine-executable instructions further comprises: identifying an operator zone within the overview image using an object locator of the operator; deleting and / or ignoring locator data and / or object position data within said operator zone; The medical system according to any one of claims 1 to 11.

13. 13. The medical system of claim 1, wherein the overview images are received as an image stream having a frame rate, the first neural network is configured to process the overview images at a rate higher than or equal to the frame rate, and execution of the machine-executable instructions causes the computing system to repeatedly input the overview images to obtain the object locator at the frame rate.

14. a first neural network, an object identification neural network, and an item identification neural network, wherein execution of the machine executable instructions provides a computing system with: receiving an overview image, said overview image having original resolution and describing a subject positioned on a subject support of a medical imaging system; receiving object locators describing positions and orientations of a plurality of predetermined objects in the overview image in response to inputting the overview image to the first neural network, the first neural network being configured to process the overview image at a first resolution, the first resolution being lower than the original resolution, the plurality of predetermined objects comprising the subject and the subject support; constructing an object sub-image for each of the plurality of predetermined objects using the object locator, each object sub-image having a second resolution that is lower than the original resolution and higher than the first resolution; receiving object position data describing each of the plurality of predetermined objects in response to inputting the object sub-image for each of the plurality of predetermined objects into the object identification neural network specific to each of the plurality of predetermined objects; constructing a plurality of search images for the plurality of predetermined objects using the object location data, the plurality of search images having a third resolution higher than the second resolution, the third resolution being lower than or equal to the original resolution; receiving locator data describing the location of an item in response to inputting each of the plurality of search images into the item identification neural network; detecting a safe condition by comparing object position data and the locator data for each of the predetermined objects against predetermined criteria; providing a warning signal if the safe condition is detected; A medical system that executes the above.

15. 1. A computer-implemented method, the method comprising: receiving an overview image, said overview image having original resolution and describing a subject positioned on a subject support of a medical imaging system; receiving object locators describing positions and orientations of a plurality of predetermined objects in the overview image in response to inputting the overview image to a first neural network, the first neural network being configured to process the overview image at a first resolution, the first resolution being lower than the original resolution, the plurality of predetermined objects comprising the subject and the subject support; constructing an object sub-image for each of the plurality of predetermined objects using the object locator, each object sub-image having a second resolution that is lower than the original resolution and higher than the first resolution; receiving object position data describing each of the plurality of predetermined objects in response to inputting the object sub-image for each of the plurality of predetermined objects into an object identification neural network that is specific to each of the plurality of predetermined objects; constructing a plurality of search images for the plurality of predetermined objects using the object location data, the plurality of search images having a third resolution higher than the second resolution, the third resolution being lower than or equal to the original resolution; receiving locator data describing the location of an item in response to inputting each of the plurality of search images into an item identification neural network; detecting a safe condition by comparing object position data and the locator data for each of the predetermined objects against predetermined criteria; providing a warning signal if the safe condition is detected; A method comprising:

Images