Generation of pseudo-radiographic images from optical images

Abstract

The present invention provides a medical instrument (100, 300) comprising a memory (110) storing machine executable instructions (120) and an image transformation neural network (122) trained to transform an optical image (124) of an object (318) into an output image (126). The output image comprises a pseudo radiographic image (600). The pseudo radiographic image is aligned with the optical image. The medical instrument further comprises a processor (104) configured to control a medical imaging system. Execution of the machine executable instructions causes the processor to: receive (200) the optical image of the object, and generate (202) the output image by inputting the optical image into the image transformation neural network.

Description

Technical Field

[0001] The present invention relates to medical imaging, and in particular to acquisition planning for medical imaging. Background Art

[0002] In various medical imaging modalities, such as magnetic resonance imaging, computed tomography, or positron emission tomography, the subject needs to be properly positioned, and in many cases the medical imaging system needs to have its region of interest adjusted to image the desired anatomical region. Typically, an initial survey scan of the subject is acquired, and the position of the subject and / or the adjustment of the region of interest is adjusted. This can be a time consuming process, and in the case of some techniques, such as computed tomography, it can expose the subject to additional ionizing radiation.

[0003] US patent application US2016 / 0073979A1 discloses determining anatomical landmarks on a patient by means of anatomical landmarks that are called up from a database with an anatomical model and converted into the individual body size and individual position of the patient. Thus, anatomical landmarks that are calculated individually for the patient and used as a piece of reference position information can be called up from the database. The positioning of the patient table is thus significantly accelerated, wherein accuracy is also improved. Thus, the piece of reference position information can be calculated individually for the same patient in different positions or for different patients with different body sizes by recalculating the piece of reference position information through a conversion rule for the respective patient. Summary of the invention

[0004] The present invention provides a medical instrument, a computer program product and a method. Embodiments are given herein.

[0005] An embodiment provides a means for generating a pseudo-radiographic image from an optical image of an object. The pseudo-radiographic image may be used, for example, in place of a survey image, or may be useful in the initial positioning of an object and / or adjustment of a region of interest. This may be achieved by acquiring an optical image (such as a normal color image, a black and white image, or even an infrared image) and inputting it into an image transformation neural network. The image transformation neural network is trained so that it outputs an output image. The output image is or includes a pseudo-radiographic image. The pseudo-radiographic image may be a representation of a real radiographic image or an idealized radiographic image.

[0006] Based on the optical image of the object alone, it would be impossible to provide a substitute for a true magnetic resonance image or a true computed tomography image. However, a pseudo radiographic image can be used to accurately indicate the location of anatomical structures within the object. This can, for example, help in planning medical imaging or radiation therapy of the object.

[0007] In one aspect, the present invention provides a medical instrument comprising a memory storing machine executable instructions and an image transformation neural network trained to transform an optical image of an object into an output image. The output image comprises a pseudo radiographic image. The pseudo radiographic image is aligned with the optical image. The medical instrument further comprises a processor configured to control a medical imaging system. Execution of the machine executable instructions causes the processor to: receive the optical image of the object, and then generate the output image by inputting the optical image into the image transformation neural network.

[0008] This embodiment may have the benefit that the normal optical image can be used to provide reliable anatomical landmarks that can be used, for example, to locate an object for medical imaging.

[0009] The image transformation neural network can be a neural network used for image processing among various types of neural networks. In one example, the image transformation neural network is a fully connected neural network with multiple hidden layers. The input to the image transformation neural network can be an image as well as an output. In another example, the image transformation neural network is a neural network with a convolutional neural network having continuous convolutional layers. The spatial dimensions of the feature map match the spatial dimensions of the input and output images. In other examples, the image transformation neural network can be a convolutional neural network with residual blocks and / or skip connections. In another example, the image transformation neural network can be an encoder-decoder type convolutional neural network. This can be equivalent to an autoencoder that has been trained to output different types of images. For example, an autoencoder can be trained to learn new representations.

[0010] As used herein, a pseudo-radiographic image is a radiographic image that has been artificially generated, such as an MRI or CT image. In different examples, the output image may take different forms. In one example, the output image may include a superposition of an optical image and a pseudo-radiographic image. In other examples, the output image may be just a pseudo-radiographic image. If the image transformation neural network is only trained to output pseudo-radiographic images, the superposition can be easily calculated using normal image processing techniques.

[0011] In some embodiments, the optical image is an optical image of a coronal plane of the object, and this is to image the surface of the object. In other examples, the optical image can also be used to image other views of the object.

[0012] In some embodiments, there is a one-to-one mapping between the input and output images. For example, the input image can have a certain size or dimension in terms of pixels or voxels. The output image can have the same size or larger so that the entire output image can be represented. In other examples, the output image can have a larger or different volume. For example, an image of the coronal plane or face or skull of an object can be used to generate a pseudo-radiographic image using three-dimensional data. In some examples, the optical image contains more information. There can be, for example, multiple cameras, 3D images, or even 3D stacks of images. Similarly, the optical image can include data views from multiple cameras or multiple slice outputs from a pseudo-radiographic image.

[0013] In another embodiment, the image transformation neural network is trained using deep learning.

[0014] In another embodiment, the image transformation neural network is a normal image transformation network formed by two convolutional neural networks.

[0015] In another embodiment, it is possible to implement that the image transformation neural network is a perceptual adversarial network. This can be particularly beneficial when training on views that are not visible to the camera.

[0016] In another embodiment, the pseudo radiographic image may be a survey-like radiographic image.In another example, the pseudo radiographic image may include a skeletal model or classification or a 3D rendering or similar animation of the anatomy of the subject.

[0017] In another embodiment, the medical instrument further comprises a medical imaging system configured to acquire radiographic images of the measurements of the object. The radiographic images may be any normal mode of radiographic images typically acquired by a medical imaging system found in a radiology department. This would include imaging techniques such as magnetic resonance imaging, computed tomography, and nuclear medicine imaging techniques such as positron emission tomography.

[0018] The medical instrument further comprises an object support configured to support the object in the medical imaging system. The medical instrument further comprises a camera configured to acquire the optical image of the object on the object support. The image acquired using the camera is registered to a coordinate system of the medical imaging system. Execution of the machine executable instructions further causes the processor to acquire the optical image of the object. Execution of the machine executable instructions further causes the processor to register the position of the object to the coordinate system of the medical imaging system using the output image.

[0019] This embodiment may be beneficial because in this embodiment, the camera acquires an optical image and the resulting output image is then used to register the position of the object. The output image includes a pseudo-radiographic image that can be used to provide a good estimate of the position of various anatomical structures within the object. This may have the benefit of facilitating imaging of the object in a medical instrument.

[0020] In another embodiment, the medical instrument further comprises a display on a user interface. Execution of the machine executable instructions further causes the processor to display the output image on the display. Execution of the machine executable instructions further causes the processor to receive position data from the user interface. The position data describes at least one position in the output image. For example, the output image can be displayed on a display for an operator or physician using a medical imaging system. Since pseudo-radiographic images essentially provide an estimate of anatomical structures within the object, the selection of position data from the user interface provides instructions on which areas of the object are to be imaged without performing a survey scan. This can, for example, help speed up the acquisition of medical images from expensive medical imaging systems (such as computed tomography or magnetic resonance imaging systems).

[0021] In another embodiment, execution of the machine executable instructions further causes the processor to receive an anatomical image. The anatomical image includes position markers. Execution of the machine executable instructions further causes the processor to register the anatomical image to the output image. Execution of the machine executable instructions further causes the processor to determine position data from the position markers by applying the registration of the anatomical image to the output image. For example, the anatomical image may be a real anatomical image, such as a CT or MR scan of a subject. The anatomical image may also be an idealized or animation type drawing used to indicate various anatomical locations on a subject.

[0022] The position markers may be coordinates or even marks or boxes drawn on the anatomical image. The anatomical image may have features that are comparable or mappable onto the pseudo-radiographic image. Since the pseudo-radiographic image indicates a specific location in the object, registration of the anatomical image identifies where the position markers are on the object. This may be useful in providing a means of identifying the location of an object to be imaged using a medical imaging system.

[0023] In another embodiment, the memory further comprises a location lookup neural network configured to receive the output image and the semantic locator location descriptor. The location lookup neural network is trained to output location data identifying the location of the semantic location descriptor in the output image. Execution of the machine executable instructions further causes the processor to receive the semantic location descriptor of an anatomical location. Execution of the machine executable instructions further causes the processor to determine location data by inputting the output image and the semantic location descriptor into the location lookup neural network.

[0024] For example, the semantic location descriptor may be audio or text data containing a semantic message. The message may then indicate a region of an object that is desired to be imaged using a medical imaging system. The location finding neural network may be trained to recognize different anatomical locations in a pseudo-radiographic image or in a normal radiographic image. This embodiment may be beneficial because it may provide a means for automatically interpreting a physician's order to image a specific region of an object.

[0025] For example, an object may be placed on an object support and then imaged using a camera. The optical image is input into an image transformation neural network and then a pseudo-radiographic image is output. A command or instruction from a physician or operator acquires a semantic location descriptor and inputs this along with the pseudo-radiographic image into a location finding neural network. The neural network is then able to identify the region and provide location data. The location data can, for example, be used to determine which part of the object is to be imaged using a medical imaging system.

[0026] The location-finding neural network may be trained by labeling regions of the pseudo-radiographic image or radiographic image and then using deep learning to train the location-finding neural network using these labels. The location-finding neural network may be, for example, a convolutional neural network.

[0027] In another embodiment, the medical imaging system is configured to acquire a radiographic image of the measurements from a region of interest. Execution of the machine executable instructions further causes the processor to determine the region of interest using the position data. Execution of the machine executable instructions further causes the processor to control the medical imaging system to acquire a radiographic image from the measurements of the object from the region of interest. This embodiment may be beneficial because control of the medical imaging system may be performed automatically once the position data is determined.

[0028] In another embodiment, the medical imaging system is a positron emission tomography system.

[0029] In another embodiment, the medical imaging system is a single photon emission tomography system.

[0030] In another embodiment the medical imaging system is a computed tomography system.

[0031] In another embodiment the medical imaging system is an X-ray machine.

[0032] In another embodiment the medical imaging system is a magnetic resonance imaging system.

[0033] In another embodiment, the camera is an optical camera. This can be beneficial, for example, because a normal optical camera can be easily used to acquire an image of the object. In some examples, the camera can be a plurality of cameras. For example, instead of having a single camera to image the object when the object is outside the medical imaging system, a larger number of smaller cameras can be used to make a composite image of the object. For example, small, inexpensive cameras such as those used in mobile phones can be distributed along the bore of the magnetic resonance imaging system. Thus, when the object is already in position, an optical image of the object can be acquired.

[0034] In another embodiment, the camera is an infrared camera. The use of an infrared camera can be beneficial because it can be used to image the body heat of the subject. This can, for example, enable effective imaging of the subject even when the subject is wearing clothing or covered. This can provide more comfort for the subject when collecting optical images.

[0035] In another embodiment, execution of the machine executable instructions further causes the processor to receive a matching pair of optical training images and radiological training images. Execution of the machine executable instructions further causes the processor to input one of the optical training images into the image transform neural network to generate a test image. There is a matching radiological training image selected from the radiological training images as a matching pair of the one of the optical training images. Execution of the machine executable instructions further causes the processor to train the image transform neural network using a difference between the test image and the matching radiological training image.

[0036] This embodiment may be beneficial because it may provide a means to efficiently train an image transformation neural network for a specific medical imaging system. This may be achieved, for example, using deep learning.

[0037] In another embodiment, the pseudo radiographic image is any one of the following: a pseudo X-ray, a pseudo magnetic resonance image, a projection of a pseudo magnetic resonance image onto a plane, a cross-section of a pseudo magnetic resonance image, a pseudo positron emission tomography image, a projection of a pseudo positron emission tomography image onto a plane, a cross-section of a pseudo positron emission tomography image, a pseudo computed tomography image, a projection of a pseudo computed tomography image onto a plane, and a cross-section of a pseudo computed tomography image.

[0038] In another aspect, the present invention provides a computer program product comprising machine executable instructions for execution by a processor controlling a medical instrument. The computer program product also includes an implementation of an image transformation neural network trained to transform an optical image of an object into an output image. The output image comprises a pseudo radiographic image. The pseudo radiographic image is aligned with the optical image. Execution of the machine executable instructions causes the processor to receive the optical image of the object. Execution of the machine executable instructions also causes the processor to generate the output image by inputting the optical image into the image transformation neural network. Advantages of the system have been discussed previously.

[0039] In another aspect, the present invention provides a medical imaging method. The method includes receiving an optical image of an object. The method also includes generating an output image by inputting the optical image into an image transformation neural network. The image transformation neural network is trained to transform the optical image of the object into the output image. The output image includes a pseudo-radiographic image. The pseudo-radiographic image is aligned with the optical image. Advantages of this method have been discussed previously.

[0040] In another embodiment, the method further comprises receiving a matching pair of optical training images and radiological training images. The method further comprises inputting one of the optical training images into the image transform neural network to generate a test image. There is a matching radiological training image selected from the radiological training images as a matching pair of the one of the optical training images. The method further comprises training the image transform neural network using a difference between the test image and the matching radiological training image.

[0041] It should be understood that one or more of the foregoing embodiments of the present invention may be combined as long as the combined embodiments are not mutually exclusive.

[0042] As will be appreciated by those skilled in the art, various aspects of the present invention may be implemented as an apparatus, method, or computer program product. Accordingly, various aspects of the present invention may take the form of a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microcode, etc.), or an embodiment of a combination of software and hardware aspects, all of which may be collectively referred to herein as a "circuit," "module," or "system." In addition, various aspects of the present invention may take the form of a computer program product implemented in one or more computer-readable media having a computer executable code implemented thereon.

[0043] Any combination of one or more computer-readable media can be utilized. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. As used herein, "computer-readable storage medium" encompasses any tangible storage medium that can store instructions that can be run by a processor of a computing device. The computer-readable storage medium can be referred to as a computer-readable non-transient storage medium. The computer-readable storage medium can also be referred to as a tangible computer-readable medium. In some embodiments, a computer-readable storage medium can also be capable of storing data that can be accessed by the processor of the computing device. Examples of computer-readable storage media include, but are not limited to: a floppy disk, a magnetic hard disk drive, a solid-state hard disk, a flash memory, a USB thumb drive, a random access memory (RAM), a read-only memory (ROM), an optical disk, a magneto-optical disk, and a register file of a processor. Examples of optical disks include compact discs (CDs) and digital versatile discs (DVDs), such as CD-ROMs, CD-RWs, CD-Rs, DVD-ROMs, DVD-RWs, or DVD-R disks. The term computer-readable storage medium also refers to various types of recording media that can be accessed by the computer device via a network or a communication link. For example, data can be retrieved via a modem, via the Internet, or via a local area network. Computer executable code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0044] Computer readable signal media may include a propagated data signal having computer executable code implemented therein, for example, in baseband or as part of a carrier wave. Such propagated signals may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. Computer readable signal media may be any computer readable medium that is not a computer readable storage medium and is capable of delivering, propagating or transmitting a program for use by or in conjunction with an instruction execution system, device or apparatus.

[0045] "Computer memory" or "memory" is an example of a computer-readable storage medium. Computer memory is any memory that is directly accessible to a processor. "Computer storage" or "storage device" is another example of a computer-readable storage medium. A computer storage device is any non-volatile computer-readable storage medium. In some embodiments, a computer storage device may also be computer memory or vice versa.

[0046] As used herein, "processor" encompasses an electronic component capable of running a program or machine executable instructions or computer executable code. References to computing devices including "processors" should be interpreted as possibly including more than one processor or processing core. The processor may be, for example, a multi-core processor. A processor may also refer to a collection of processors within a single computer system or distributed among multiple computer systems. The term computing device should also be interpreted as possibly referring to a collection or network of computing devices, each of which includes one or more processors. The computer executable code may be run by multiple processors, which may be within the same computing device or may even be distributed across multiple computing devices.

[0047] The computer executable code may include machine executable instructions or programs that cause a processor to perform aspects of the present invention. The computer executable code for performing operations for various aspects of the present invention may be written in any combination of one or more programming languages ​​and compiled into machine executable instructions, including object-oriented programming languages ​​such as Java, Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the C programming language or similar programming languages. In some instances, the computer executable code may take the form of a high-level language or a precompiled form and be used in conjunction with an interpreter that generates running machine executable instructions.

[0048] The computer executable code may be run entirely on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server as a stand-alone software package. In the latter case, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or a connection may be made to an external computer (e.g., through the Internet using an Internet service provider).

[0049] Various aspects of the present invention are described with reference to the flowchart illustration and / or block diagram of the method, device (system) and computer program product according to an embodiment of the present invention.It will be understood that each frame or the part of frame of flow chart, diagram and / or block diagram can be implemented by the computer program instruction in the form of computer executable code when applicable.It should also be understood that when not mutually exclusive, the combination of frames in different flow charts, diagrams and / or block diagrams can be combined.These computer program instructions can be provided to the processor of general-purpose computer, special-purpose computer or other programmable data processing device to produce machine, so that the instruction run by the processor of computer or other programmable data processing device creates the module for implementing the function / action specified in one or more frames of flow chart and / or block diagram.

[0050] These computer program instructions may also be stored in a computer-readable medium, which is capable of directing a computer, other programmable data processing apparatus, or other device to work in a specific manner, so that the instructions stored in the computer-readable medium produce a product including instructions for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0051] The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operating steps to be performed on the computer, other programmable apparatus, or other devices to produce a computer-implemented process, such that the instructions running on the computer or other programmable apparatus provide a process for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0052] As used in this article, "user interface" is an interface that allows a user or operator to interact with a computer or computer system. "User interface" may also refer to "human-machine interface device". A user interface may provide information or data to an operator, and / or receive information or data from an operator. A user interface may enable the input from an operator to be received by a computer, and the output from the computer may be provided to the user. In other words, a user interface may allow an operator to control or manipulate a computer, and the interface may allow the computer to indicate the effect of the control or manipulation of the operator. The display of data or information on a display or graphical user interface is an example of providing information to an operator. The reception of data by a keyboard, mouse, tracking ball, touch pad, pointing stick, graphic input board, joystick, game pad, webcam, headgear, pedal, wired gloves, remote control, and accelerometer is all an example of a user interface component that realizes receiving information or data from an operator.

[0053] As used herein, "hardware interface" encompasses an interface that enables a processor of a computer system to interact with or control an external computing device and / or apparatus. A hardware interface may allow a processor to send a control signal or instruction to an external computing device and / or apparatus. A hardware interface may also enable a processor to exchange data with an external computing device and / or apparatus. Examples of hardware interfaces include, but are not limited to, a universal serial bus, an IEEE 1394 port, a parallel port, an IEEE 1284 port, a serial port, an RS-232 port, an IEEE-488 port, a Bluetooth connection, a wireless LAN connection, a TCP / IP connection, an Ethernet connection, a control voltage interface, a MIDI interface, an analog input interface, and a digital input interface.

[0054] As used herein, a "display" or "display device" encompasses an output device or user interface suitable for displaying images or data. A display may output visual, audio, and / or tactile data. Examples of displays include, but are not limited to: computer monitors, television screens, touch screens, tactile electronic displays, Braille screens, cathode ray tubes (CRTs), memory tubes, bi-stable displays, electronic paper, vector displays, flat panel displays, vacuum fluorescent displays (VFs), light emitting diode (LED) displays, electroluminescent displays (ELDs), plasma display panels (PDPs), liquid crystal displays (LCDs), organic light emitting diode displays (OLEDs), projectors, and head mounted displays.

[0055] Magnetic resonance (MR) data is defined herein as the measurement results recorded using the antenna of a magnetic resonance device of radio frequency signals emitted by atomic spins during a magnetic resonance imaging scan. MRF magnetic resonance data is magnetic resonance data. Magnetic resonance data is an example of medical imaging data. A magnetic resonance imaging (MRI) image or MR image is defined herein as a reconstructed two-dimensional or three-dimensional visualization of anatomical data contained within the magnetic resonance imaging data. This visualization can be performed using a computer. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In the following, preferred embodiments of the invention will be described by way of example only and with reference to the accompanying drawings, in which:

[0057] Figure 1 Examples of medical instruments are illustrated;

[0058] Figure 2 The diagram shows the operation Figure 1 A flowchart of a method of a medical instrument;

[0059] Figure 3 Another example of a medical instrument is shown;

[0060] Figure 4 Shows Figure 3 Another view of a medical instrument;

[0061] Figure 5 The diagram shows the operation Figure 3 A flowchart of a method of a medical instrument;

[0062] Figure 6 illustrates an example of a pseudo-radiographic image; and

[0063] Figure 7 An example of a method for training an image transformation neural network is shown.

[0064] Reference numerals list

[0065] 100 Medical instruments

[0066] 102 Computer

[0067] 104 Processor

[0068] 106 Hardware Interface

[0069] 108 User Interface

[0070] 110 Memory

[0071] 120 machine executable instructions

[0072] 122 Image Transformation Neural Network

[0073] 124 Optical images

[0074] 126 output images (including pseudo-radiographic images)

[0075] 130 optical training images

[0076] 132 Radiology training images

[0077] 134 test images

[0078] 136 Differences between radiology training and test images

[0079] 200 receiving an optical image of an object

[0080] 202 Generate an output image by inputting an optical image into an image transformation neural network

[0081] 300 Medical Imaging Systems

[0082] 302 Magnetic Resonance Imaging System

[0083] 304 Magnet

[0084] 306 Magnet bore

[0085] 308 Imaging Area

[0086] 309 Area of ​​Interest

[0087] 310 Magnetic Field Gradient Coil

[0088] 312 Magnetic Field Gradient Coil Power Supply

[0089] 314 RF Coil

[0090] 316 Transceiver

[0091] 318 objects

[0092] 320 Object Support

[0093] 322Actuator for object support

[0094] 324 Camera

[0095] 330 Pulse sequence command

[0096] 332 Location data

[0097] 334 Location of the area of ​​interest

[0098] 336 anatomical images

[0099] 338 Registration of anatomical images to output images

[0100] 340 Semantic Location Descriptor

[0101] 342 Location Finding Neural Network

[0102] 344 Magnetic Resonance Data

[0103] 346 Magnetic resonance images

[0104] 500 controls the camera to collect the optical image of the object

[0105] 502 Using the output image to register the position of the object to the coordinate system of the medical imaging system

[0106] 504 Determining Position Data Using Pseudo-Radiographic Images

[0107] 506 Using location data to determine area of ​​interest

[0108] 508 controlling the medical imaging system to acquire a measured radiographic image from the object from the region of interest

[0109] 600 pseudo-radiological images

[0110] 700 Profiles from Oncologists

[0111] 702 Infrared Image

[0112] 704 Input

[0113] 706 Deep Learning Algorithms

[0114] 708 Output

[0115] 710 Training

[0116] 712 Expected value

[0117] 716 Measured table movement

[0118] Scan volume at the isocenter of the 718 scanner DETAILED DESCRIPTION

[0119] Like numbered elements in these figures are equivalent elements or perform the same function. If the function is equivalent, it will not be necessary to discuss an element that has been discussed previously in a later figure.

[0120] Figure 1 An example of a medical instrument 100 is illustrated. Figure 1 The medical instrument 100 in the embodiment includes a computer 102. The computer 102 includes a processor 104, which communicates with an optional hardware interface 106, an optional user interface 108, and a memory 110. The memory 110 can be any combination of memories accessible to the processor 104. This can include, for example, main memory, cache memory, and non-volatile memory (such as flash RAM, hard drive or other storage device). In some examples, the memory 104 can be considered as a non-transitory computer readable medium. The hardware interface 106 can, for example, be used to communicate with other computer systems and / or control other components of the medical instrument 100. The user interface 108 can, for example, be used to display data and / or to receive commands or other information from a user or operator.

[0121] The memory 110 is shown as containing machine executable instructions 120. The machine executable instructions 120 contain commands that enable the processor 104 to control the medical instrument 100. The machine executable instructions 120 may also control other commands that enable the processor 104 to perform various types of image processing.

[0122] The memory 110 is also shown as containing an image transform neural network 122. The image transform neural network 122 takes an image as input and outputs another image. The image transform neural network 122 has been trained to take an image of an object and output a pseudo radiological image. The memory 110 is shown as containing an optical image 124 of the object. The memory 110 is also shown as containing an output image, which includes a pseudo radiological image obtained by inputting the optical image 124 into the image transform neural network 122. In some instances, the output image is simply a pseudo radiological image.

[0123] The memory 110 is shown as containing an optional optical training image 130. The memory 110 is also shown as containing an optional radiology training image 132. The optical training image 130 and the radiology training image 132 can be considered a matching pair. They are both obtained from an object when the object is on a support. For example, the optical training image 130 can be an image of an object that is to be placed on an object support within a magnetic resonance imaging system, and the radiology training image 132 can be an image of the object acquired by the magnetic resonance imaging system. The memory 110 also contains a test image 134 obtained by inputting the optical training image 130 into the image transform neural network 122. The test image 134 and the radiology training image 132 are then used to generate or calculate a difference 136 between the two images. The difference 136 is then used to further train the image transform neural network 122. For example, this can be performed using deep learning.

[0124] Figure 2 The diagram shows the operation Figure 1 Flowchart of a method of a medical instrument 100. First, in step 200, an optical image 124 is received. The reception of the optical image can be performed in different ways. In one example, there can be a camera, and the reception of the optical image includes capturing the image using the camera. In other examples, receiving the optical image can include accessing the optical image from the memory 110, obtaining the optical image via the network connection 106, or from other data carrying media. The method then proceeds to step 202. In step 202, an output image 126 is generated by inputting the optical image 124 into the image transformation neural network 122.

[0125] Figure 3 Yet another example of a medical apparatus 300 is illustrated. The medical apparatus 300 is shown to additionally include a magnetic resonance imaging system 302 and a camera 324. In this example, the magnetic resonance imaging system 302 is used, however, it is intended to be illustrative and the magnetic resonance imaging system 302 may be replaced with other medical imaging systems such as a computed tomography system or a positron emission tomography system.

[0126] Magnetic resonance imaging system 302 includes magnet 304. Magnet 304 is a superconducting cylindrical magnet with a bore 306 passing through it. The use of different types of magnets is also possible, for example, it is also possible to use both a split cylindrical magnet and a so-called open magnet. A split cylindrical magnet is similar to a standard cylindrical magnet, except that the cryostat has been divided into two parts to allow access to the isoplane of the magnet, such a magnet can be used, for example, in combination with charged particle beam therapy. An open magnet has two magnet parts, one above the other, with a large enough space between them to receive an object: the arrangement of the two-part area is similar to the arrangement of Helmholtz coils. Open magnets are popular because the object is less constrained. There is a collection of superconducting coils inside the cryostat of the cylindrical magnet. Within the bore 306 of the cylindrical magnet 304, there is an imaging zone 308 where the magnetic field is strong enough and uniform enough to perform magnetic resonance imaging.

[0127] Also within the bore 306 of the magnet is a collection of magnetic field gradient coils 310 for acquiring primary magnetic resonance data to spatially encode magnetic spins within an imaging zone 308 of the magnet 304. The magnetic field gradient coils 310 are connected to a magnetic field gradient coil power supply 312. The magnetic field gradient coils 310 are intended to be representative. Typically, the magnetic field gradient coils 310 include three separate sets of coils for spatially encoding in three orthogonal spatial directions. The magnetic field gradient power supply supplies current to the magnetic field gradient coils. The current supplied to the magnetic field gradient coils 310 is controlled according to time and can be ramped or pulsed.

[0128] Adjacent to the imaging zone 308 is a radio frequency coil 314 for manipulating the orientation of magnetic spins within the imaging zone 308 and for receiving radio transmissions from spins also within the imaging zone 308. The radio frequency antenna may include multiple coil elements. The radio frequency antenna may also be referred to as a channel or antenna. The radio frequency coil 314 is connected to a radio frequency transceiver 316. The radio frequency coil 314 and the radio frequency transceiver 316 may be replaced by separate transmitting and receiving coils and separate transmitters and receivers. It should be understood that the radio frequency coil 314 and the radio frequency transceiver 316 are representative. The radio frequency coil 314 is intended to also represent a dedicated transmitting antenna and a dedicated receiving antenna. Similarly, the transceiver 316 may also represent a separate transmitter and receiver. The radio frequency coil 314 may also have multiple receiving / transmitting elements, and the radio frequency transceiver 316 may have multiple receiving / transmitting channels. For example, if a parallel imaging technique (such as SENSE) is performed, the radio frequency coil 314 will have multiple coil elements.

[0129] 104. The transceiver 316 and the gradient controller 312 are shown connected to a hardware interface 328 of a computer system 326. An object 318 is shown outside the bore 306 of the magnet 304. An object support 320 is connected to an actuator 322 for moving the object support 320. The actuator 322 enables the object 318 to be moved into and out of the bore 306 of the magnet 304. In the current position, the object 318 is positioned below a camera 324. The camera 324 can be controlled by the processor 104 to acquire the optical image 124.

[0130] Once the optical image is acquired, the optical image 124 is input into the image transformation neural network 122 to generate an output image 126 that includes or is a pseudo-radiographic image. Position data 332 is derived from the pseudo-radiographic image 126. This can be performed in different ways. For example, the user interface 108 can display the output image 126 and receive input data, which is then used to derive the position data. For example, an operator or physician can place a mark on the output image to indicate where the region of interest should be. The position of the camera 324 can be registered to the coordinate system of the magnetic resonance imaging system.

[0131] The position data 332 can then be used to directly determine the position of the region of interest 334. There are also several alternatives. For example, the processor 104 can receive an anatomical image 336, which is shown as being stored in the memory 110. The anatomical image 336 can include an image or a representation of an image marked with a desired region of interest. The anatomical image 336 can then be registered 338 to the output image to produce a registration 338 of the anatomical image to the output image. This registration 338 enables the markings on the anatomical image 336 to be used to determine the position of the region of interest 334. In another example, a semantic position descriptor 340 can be received. This can be, for example, a verbal or textual command to execute a specific imaging protocol or a description of a specific anatomical region to be imaged. The semantic position descriptor 340 and the pseudo-radiographic image 126 can be input into a position-finding neural network 342, which then outputs the position of the region of interest 334. The memory is also shown as containing a pulse sequence command 330. The pulse sequence command 330 enables the processor 104 to acquire magnetic resonance data from the object 318 at the region of interest.

[0132] Figure 4 Shows Figure 3308, except that the object 318 has now been moved into the bore 306 of the magnet 304 by the actuator 322. The region of interest 309 can be seen as being positioned within the imaging zone 308. The position of the region of interest is determined using the position 334. The memory 110 is also shown as containing magnetic resonance data 344, which was acquired using the pulse sequence commands 334 to acquire data from the region of interest 309. The memory 110 is also shown as containing a magnetic resonance image 346 reconstructed from the magnetic resonance data 344.

[0133] Figure 5 The diagram shows the operation Figure 3 and 4 Flowchart of the method of the medical instrument 300 illustrated in FIG. First, in step 500, the processor 104 controls the camera 324 to collect the optical image 124. Next, in step 200, the optical image 124 is received by the processor. Next, in step 202, the processor generates an output image 126 or a pseudo radiographic image by inputting the optical image 124 into the image transformation neural network 122. Next, in step 502, the position of the object 318 is registered to the coordinate system of the medical imaging system or magnetic resonance imaging system 302 using the output image 126. Next, in step 504, the position data 322 is determined using the output image 126. This can be implemented in any manner previously discussed, such as displaying the output image on a user interface and using a marker on this to determine the position data. Next, in step 506, the region of interest 309 is determined using the position data 332. The position can be stored in a memory as a position 334, for example, as shown in the figure. Finally, in step 508, the medical imaging system 302 is controlled by the processor to collect a measured radiographic image 346 from the region of interest 309. In this particular example, the measured radiographic image is a magnetic resonance image 346 .

[0134] Currently, the state of the art in radiation oncology simulation scanning is to use a survey scan to position the main scan field of view (region of interest 309) over the subject 318. The instructions for positioning the scan volume typically come in the form of a two-dimensional drawing of the patient's anatomy on which the oncologist has drawn the region of interest 309.

[0135] In some cases, the outline of the patient (acquired via an external camera 324, such as an infrared camera) can be used to position the imaging volume even when a patient cover partially obscures the patient.

[0136] With a survey scan, additional time is spent or additional radiation is given to the patient. Patient contours suffer from inaccuracies because bone-based volume positioning is error-prone and difficult.

[0137] Some examples may superimpose a bone model (pseudo-radiographic image 600) fused with (e.g., on top of) the patient outline 124, or show the bone instead of the patient outline. This aids in manual positioning of the scan volume 309. Another example may display a different organ, such as a soft tissue organ, fused with the patient outline.

[0138] Another example uses positioning input from an oncologist (or other operator) to move the patient into the scanner isocenter and automatically positions the positioning scan volume to cover the input volume. The superimposed organ and / or bone model can still be shown with the patient outline for user confirmation.

[0139] In one example, a method is provided for training an algorithm to adapt to the skeleton (600) inside the contours of a patient based on previous photo-survey scan pairs (130, 132) acquired using conventional simulation scans using, for example, machine learning. As a result of the training, the algorithm is able to generate survey-like pictures when presented with the photo.

[0140] The generation of training data can be arranged using an infrared camera 324 (or an optical camera) positioned above the patient 318. Images 130 can be acquired with the initial patient positioning (without mechanical obstructions, such as MR coils). The resulting survey scans 132 are stored to form a photograph-survey scan pair. Conventional survey scans can be simplified into two-dimensional images that are easily matched to two-dimensional patient photographs using maximum intensity projections (MIP). Due to the training, the algorithm is able to generate MIP images when presented with photographs.

[0141] In another example, a single photo is replaced with photos from multiple directions for redundancy / stereoscopic vision with an improved skeletal model. Using the stereo information, an improved 3D skeletal model can be generated for 3D volume positioning. In this case, a deep learning network that can match a 3D survey or actual scan with the 3D photo data can be used. The scan can be simplified by thresholding or segmenting the survey volume to show only the target organ or bone. Due to training, the algorithm is able to generate a 3D representation of the organ or bone when presented with a stereo photo.

[0142] Figure 6 An example of an output image 126 is illustrated. The output image 126 includes a ghost or protrusion of the original optical image 124. The pseudo radiological image 600 is superimposed on the original optical image 124. Using the output image 126, a radiologist or other technician can better infer various anatomical locations within the subject. Figure 6In the example of FIG. 1 , the output image 126 is placed on the display and the operator or radiologist drags the box 332. In this case, the box 332 represents position data. The position data 332 indicates the area of ​​the object that the operator or radiologist wishes to image or scan.

[0143] Figure 7 One method of training an image transformation neural network 122 is schematically shown. In this example, an infrared image 702 is used as an optical training image 130; a contour from an oncologist 700 is used as a radiology training image 132. Both are input 704 into a deep learning algorithm 706. The output of the deep learning algorithm 706 is output 708 into a training routine 710. The training routine 710 also obtains a plurality of expected values ​​712. These may be, for example, a measured table movement 714 and an expected scan volume 716 on the scanner isocenter. The scan volume 716 may correspond to the region of interest 309.

[0144] like Figure 7 , the enclosed skeletal model 132 and the photograph(s) 130 are fed as inputs 704 into a training algorithm 706. The algorithm is trained to produce outputs for table movement (if photographed outside the scanning bore) and scan volume coordinates 714 that match the user generated values ​​that had been recorded when the training data was generated.

[0145] While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the invention is not limited to the disclosed embodiments.

[0146] Those skilled in the art will be able to understand and implement other variations of the disclosed embodiments when practicing the claimed invention by studying the drawings, the specification and the claims. In the claims, the word "comprising" does not exclude other elements or steps, and the words "one" or "an" do not exclude multiple. A single processor or other unit may perform the functions of several items recited in the claims. The recording of specific elements in mutually different dependent claims does not indicate that a combination of these elements cannot be used to advantage. The computer program may be stored and / or distributed on an appropriate medium, such as an optical storage medium or solid-state medium supplied with or as part of other hardware, but the computer program may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. Any reference numerals in the claims shall not be interpreted as limiting the scope.

Claims

1. A medical instrument (100, 300), comprising: - a medical imaging system (302) configured to acquire measured radiographic images (346) of a subject; - a subject support (320) configured to support the subject in the medical imaging system; a camera (324) configured to acquire optical images of the object on the object support, wherein the images acquired using the camera are registered to a coordinate system of the medical imaging system; a memory (110) storing machine executable instructions (120) and an image transformation neural network (122) trained to transform an optical image (124) of the object into an output image (126), wherein the output image comprises a pseudo radiographic image (600), wherein the pseudo radiographic image is aligned with the optical image; and - a processor (104) configured to control the medical imaging system, wherein execution of the machine executable instructions causes the processor to: - controlling (500) the camera to capture the optical image of the object; - receiving (200) said optical image of said object; - generating (202) the output image by inputting the optical image into the image transformation neural network; and - Registering (502) the position of the object to the coordinate system of the medical imaging system using the output image.

2. The medical instrument according to claim 1, wherein: The medical instrument further comprises a display and a user interface (108), wherein execution of the machine-executable instructions further causes the processor to: - displaying the output image on the display; and - receiving position data from the user interface, wherein the position data describes at least one position in the output image.

3. The medical instrument according to claim 1, wherein: Execution of the machine executable instructions further causes the processor to: - receiving an anatomical image (336), wherein the anatomical image includes position markers; - calculating a registration by registering the anatomical image to the pseudo-radiographic image (338); and - determining (504) position data from said position markings by applying a registration of said anatomical image to said output image.

4. The medical instrument according to claim 1, wherein: The memory further comprises a location lookup neural network (342) configured to receive the output image and the semantic location descriptor (340), wherein the location lookup neural network is trained to output location data identifying the location of the semantic location descriptor in the output image; Wherein, execution of the machine executable instructions further causes the processor to: - receiving said semantic location descriptor of an anatomical location; and - determining (504) location data by inputting the output image and the semantic location descriptor into the location lookup neural network.

5. The medical instrument according to claim 2, 3 or 4, wherein: The medical imaging system is configured to acquire the measured radiographic image from a region of interest (309), wherein execution of the machine-executable instructions further causes the processor to: - using said location data to determine (506) said region of interest; and - controlling (508) the medical imaging system to acquire radiographic images from the measurements of the subject from the region of interest.

6. The medical instrument according to any one of claims 1 to 4, wherein: The medical imaging system is any one of: a positron emission tomography system, a single photon emission tomography system, a computed tomography system, an X-ray machine, and a magnetic resonance imaging system (302).

7. The medical instrument according to any one of claims 1 to 4, wherein: The camera is any one of the following: an optical camera and an infrared camera.

8. The medical instrument according to any one of claims 1 to 4, wherein: The output image is used in place of a survey image to register the position of the object to the coordinate system of the medical imaging system using the output image.

9. The medical instrument according to any one of claims 1 to 4, wherein: Execution of the machine executable instructions further causes the processor to: - receiving a matching pair of an optical training image (130) and a radiological training image (132); - inputting one of the optical training images (130) into the image transform neural network to generate a test image (134), wherein there is a matching radiological training image (132) selected from the radiological training images as a matching pair of the one of the optical training images; and - using the difference (136) between the test image and the matched radiology training image to train the image transformation neural network.

10. The medical instrument according to any one of claims 1 to 4, wherein: The pseudo-radiographic image is any of the following: a pseudo-X-ray, a pseudo-magnetic resonance image, a projection of a pseudo-magnetic resonance image onto a plane, a cross-section of a pseudo-magnetic resonance image, a pseudo-positron emission tomography image, a projection of a pseudo-positron emission tomography image onto a plane, a cross-section of a pseudo-positron emission tomography image, a pseudo-computed tomography image, a projection of a pseudo-computed tomography image onto a plane, and a cross-section of a pseudo-computed tomography image.

11. The medical instrument according to any one of claims 1 to 4, wherein: Any of: the output image is the pseudo radiographic image, and the output image is a superposition of the optical image on the pseudo radiographic image.

12. A computer program product comprising machine executable instructions (120) for execution by a processor (104) for controlling a medical instrument (100, 300), wherein: The medical instrument comprises a medical imaging system (302) configured to acquire a measured radiographic image (346) of an object, wherein the medical instrument further comprises a subject support (320) configured to support the object in the medical imaging system, wherein the medical instrument further comprises a camera (324) configured to acquire an optical image of the object on the subject support, wherein the image acquired using the camera is registered to a coordinate system of the medical imaging system; wherein the computer program product further comprises an implementation of an image transformation neural network (122) trained to transform the optical image (124) of the object into an output image (126), wherein the output image comprises a pseudo radiographic image (600), wherein the pseudo radiographic image is aligned with the optical image, and wherein execution of the machine executable instructions causes the processor to: - controlling (500) the camera to capture the optical image of the object; - receiving (200) said optical image of said object; - generating (202) the output image by inputting the optical image into the image transformation neural network; and - Registering (502) the position of the object to the coordinate system of the medical imaging system using the output image.

13. A medical imaging method, wherein: The method comprises: - controlling (500) a camera to acquire an optical image of an object, wherein the camera (324) is configured for acquiring the optical image of the object on an object support of a medical imaging system, wherein the medical imaging system (302) is configured for acquiring measured radiographic images (346) of the object, wherein the object support (320) is configured for supporting the object in the medical imaging system, wherein the image acquired using the camera is registered to a coordinate system of the medical imaging system; - receiving (200) an optical image (124) of an object; and - generating (202) an output image (126) by inputting the optical image into an image transformation neural network (122), wherein the image transformation neural network is trained to transform the optical image of the object into the output image, wherein the output image comprises a pseudo radiographic image (600), wherein the pseudo radiographic image is aligned with the optical image; and - Registering (502) the position of the object to the coordinate system of the medical imaging system using the output image.

14. The medical imaging method according to claim 13, wherein: The method further comprises: - receiving a matching pair of an optical training image (130) and a radiological training image (132); - inputting one of the optical training images (130) into the image transform neural network to generate a test image (134), wherein there is a matching radiological training image (132) selected from the radiological training images as a matching pair of the one of the optical training images; and - using the difference (136) between the test image and the matched radiology training image to train the image transformation neural network.

Images