Virtual fiducial marking for automated planning in medical imaging

Abstract

Disclosed herein are medical systems 100, 300, 500, 600 comprising a medical imaging system 102, 302 configured to acquire medical imaging data 136 depicting a subject 100, a camera system 114 configured to acquire a subject image 138 of the subject, and a memory 126 storing machine executable instructions 130, medical imaging system commands 134, coordinate system mapping, and an image marking neural network 132. Execution of the machine-executable instructions by the computing system 120 causes the computing system to acquire medical imaging data (200) by controlling the medical imaging system with medical imaging system commands, repeatedly control the camera system to acquire subject images during acquisition of the medical imaging data (202), repeatedly receive camera system coordinates 142 of virtual reference markers by inputting the subject images to an image marking neural network (204), and repeatedly provide imaging system coordinates of a set of virtual reference markers (206) by repeatedly converting the camera system coordinates of the virtual reference markers into the provided imaging system coordinates of the virtual reference markers using coordinate system mapping.

Description

[Technical field]

[0001] The present invention relates to the use of fiducial markings in medical imaging. [Background technology]

[0002] Various tomographic medical imaging techniques, such as magnetic resonance imaging (MRI), computed tomography, positron emission tomography, and single photon emission tomography, allow detailed visualization of a subject's anatomy. A common feature of all these imaging modalities is the significant time required to acquire the medical imaging data required to reconstruct a medical image. During acquisition of the medical imaging data, the subject moves, either voluntarily or involuntarily, resulting in image corruption or artifacts. The subject's motion can be tracked to reduce the effects of the subject's motion. In the case of predictive motion correction, the medical imaging system is adjusted to compensate for the subject's motion during acquisition of the medical imaging data. Retrospective motion correction is another class of technique, in which the subject's motion is recorded during acquisition of the medical imaging data, and the medical imaging data is corrected using the recorded subject's motion to reconstruct a corrected medical image.

[0003] A well-known technique is the use of fiducial markers. A fiducial marker is a marker that is applied to the subject's surface. In some instances, a camera can then be used to measure the position of the fiducial marker. In other instances, the position of the fiducial marker is directly recorded by a medical imaging scanner. In magnetic resonance imaging, there are fiducial markers that can be directly detected and located by magnetic resonance imaging measurements. A drawback of fiducial markers is that they must be applied or worn on the subject's surface.

[0004] The journal article "Markerless motion tracking and correction for PET, MRI, and simultaneous PET / MRI" by Slipsager, Jakob M. et al., Plos one 14.4 (2019): e0215524, discloses a markerless motion tracker compatible with PET, MRI, and simultaneous PET / MRI systems for motion correction (MC) of brain imaging. Compatibility with PET and MRI is achieved by placing a vision extender inside the bore and all electronic components outside the bore. The motion tracker is demonstrated in a clinical setup during a pediatric PET / MRI study involving scans of 94 pediatric patients. PET MC is presented for two of these scans using a customized version of the multiple acquisition frame method. Predictive MC of MRI acquisitions of two healthy subjects is demonstrated using a motion-aware MRI sequence. Real-time motion estimation is accompanied by a tracking validity parameter to improve tracking reliability.

[0005] I. Frosio et al., in "A Neural Network Based Method for Optical Patient Set-up Registration in Breast Radiotherapy", Annals of Biomedical Engineering, vol. 34, no. 4, pp. 677-686 (2006), disclose a radiotherapy system having a motion-compensated neural network algorithm that analyzes the position of a marker formed by a laser spot on a patient. Summary of the Invention

[0006] The invention provides a medical system, a computer program and a method in the independent claims. Embodiments are set out in the dependent claims.

[0007] The embodiments provide an improved means of tracking a subject during acquisition of medical imaging data by a medical imaging system. Instead of placing fiducial markers on the subject's surface, an image marking neural network is used to provide a set of virtual fiducial markers in the coordinate system of the camera system. Coordinate system mapping is then used to transform the positions of the set of virtual fiducial markers to the coordinate system of the medical imaging system. This has various potential advantages. First, the application of fiducial markers to the subject is avoided. Another potential advantage is that many algorithms and techniques that use physical fiducial markers can be used with minimal modification. The coordinates of the virtual fiducial markers can be fed into these existing algorithms.

[0008] Another potential advantage is that conventional systems using various artificial intelligence (AI) systems, such as the Slipsinger paper cited above, use neural networks to directly provide rigid body translation. This is effective for imaging areas such as the brain, where the organs are located within the skull. However, the techniques disclosed in Slipsinger are generally not applicable to other types of imaging. Providing virtual fiducial markers instead can increase the likelihood of dealing with various types of motion. For example, chest expansion and contraction due to breathing and cardiac movement by the subject can be handled by fiducial markers, but not by rigid body translation.

[0009] In Slipsinger et al., the TCL2 motion tracking software package (TracSuite) estimates the patient's head pose using an Iterative Closest Point (ICP) algorithm. Head pose is calculated at a frequency of 30 Hz, a sufficient sample rate for most motion correction purposes. The ICP registration algorithm imposes the assumption of tracking a rigid surface. Occlusion, rapid movements, or facial translation violate such rigidity assumptions. Also, accurate tracking becomes impossible if the patient moves outside the field of view (FOV). Therefore, the estimated pose is analyzed against a reference point cloud to determine how reliable the tracking results are.

[0010] In the present invention, the virtual reference markers as trackers are a set of movable markers that eliminate the dependency on registering the point cloud with the patient, which has the advantage of removing the obstacle of degree of freedom dependence, and makes the system more robust with three-way correction, i.e., helps to inform the patient about the patient's movement, retroactive correction and prospective correction.

[0011] In one aspect, the invention provides a medical system comprising a medical imaging system configured to acquire medical imaging data depicting a subject from an imaging zone. The medical imaging system is for example a tomographic medical imaging system. The medical imaging system has an image system coordinate system, i.e. the data is acquired with respect to a set of coordinates belonging to the medical imaging system. The medical system further comprises a camera system configured to acquire subject images of the subject during acquisition of the medical imaging data. The camera system has a camera coordinate system.

[0012] In various examples, the camera system coordinates have different meanings depending on the type of camera system used, e.g., an optical camera system or an infrared camera system. In general, the camera system can be implemented using most 3D or 2D camera systems that may be compatible with the particular type of medical imaging system being used.

[0013] In the case of a two-dimensional camera, the camera coordinates are, for example, according to the position on the image. In another example, the camera system is a three-dimensional camera or a stereo camera. In that case, the image in the three-dimensional image indicates the position in space. A stereo camera is constructed from two two-dimensional cameras. In the case where the medical imaging system is a combination of a magnetic resonance imaging (MRI) system and a positron emission tomography (PET) system, the camera is a three-dimensional camera that is compatible with both magnetic resonance imaging and positron emission tomography. This involves positioning the camera so as not to interfere with the measurements by the PET ring.

[0014] The medical system further comprises a memory storing the machine executable instructions. The memory further stores the medical imaging system commands. The memory further stores the coordinate system mapping. The memory further stores the image marking neural network. The coordinate system mapping is a mapping between the system coordinate system and the camera coordinate system. This can be implemented in various ways in various examples, for example. In one example, the coordinate system mapping is a lookup table or equation to calculate between the two.

[0015] An image marking neural network is a neural network configured to receive an input image depicting a given anatomical region of a subject. Thus, an image marking neural network is a neural network configured to locate an object in an image or to segment an image. Various types of neural networks are used. In medical imaging, neural networks with U-net architecture are often used for image segmentation and are useful for implementing image marking neural networks.

[0016] In one example, a combination of U-Net architecture and keypoint detection neural network is used to form more robust predictions for virtual fiducial markers. This network has three different blocks, similar to Mask-RCNN: segmentation, regression, and keypoint location / selection (selection of virtual fiducial markers). This helps to further increase robustness in selecting alternative points (alternate fiducial markers) when patient motion may occur. By incorporating this redundancy into the network based on segmentation, keypoint detection, and landmark identification / selection, the algorithm remains flexible and can operate with alternative fiducial points as the patient moves.

[0017] The image marking neural network is further configured to output camera system coordinates of the set of virtual reference markers, the coordinates of the set of virtual reference markers being given with respect to the input image and being created in response to receiving the input image, in other words, the input image is input to the image marking neural network and as an output the coordinates of the set of virtual reference markers are provided in the camera coordinate system.

[0018] The medical imaging system commands are configured to control the medical imaging system to acquire medical imaging data. These medical imaging system commands may, for example, take different forms in different examples. If the medical imaging system is a magnetic resonance imaging system, the medical imaging system commands are pulse sequence commands. The medical system further comprises a computing system configured to control the medical system.

[0019] Execution of the machine-executable instructions causes the computing system to acquire medical imaging data by controlling the medical imaging system with medical imaging system commands. Execution of the machine-executable instructions further causes the computing system to repeatedly control the camera system to acquire subject images during acquisition of the medical imaging data. Execution of the machine-executable instructions further causes the computing system to repeatedly receive camera system coordinates of the virtual reference markers by inputting the subject images to an image marking neural network. Execution of the machine-executable instructions further causes the computing system to repeatedly provide imaging system coordinates of a set of virtual reference markers by repeatedly transforming the camera system coordinates of the virtual reference markers using a coordinate system mapping into the provided imaging system coordinates of the virtual reference markers.

[0020] In other words, when medical imaging data is acquired, a camera acquires an image or subject image during acquisition of this medical imaging data, which is then input to an image marking neural network, which creates coordinates of a set of virtual reference markers in the coordinate system of the camera system, and coordinate system mapping is then used to transform these coordinates into the coordinate system of the medical imaging system.

[0021] The use of fiducial markers is common in various medical imaging modalities, especially in magnetic resonance imaging. A well-known drawback of the use of fiducial markers is the need to apply them to the surface or clothing of the subject to be imaged, and then to detect the position of these fiducial markers. The embodiments have the advantage that it is possible to provide the coordinates of virtual fiducial markers in the coordinate system of the medical imaging system without actually using fiducial markers. This allows existing algorithms and programs, for example for compensating for subject motion, to be used without significant modification.

[0022] The virtual reference markers take different forms in different examples. In some cases, the virtual reference markers are markers that are identified as having a specific location in the image. They are located by the image marking neural network. The term "image marking" in the image marking neural network is a label to identify the specific neural network. In one example, the set of the set of virtual reference markers is a set of point clouds detected around eye contact, corners of the mouth, or facial contours. In another example, they are various anatomical landmarks visible on the subject's surface. In another example, the set of virtual reference markers has a location determined by a relationship between multiple anatomical landmarks.

[0023] The image marking neural network can be trained in a variety of ways. Images of the subject can be prepared and marked up manually or by machine to indicate the location of the fiducial markers. The image marking neural network can then be trained using, for example, a deep learning training algorithm.

[0024] In one example, a neural network is trained with existing images based on an automatic segmentation algorithm to predict automatic 3D depth segment labels of organs and leverage previous similar labels in the field. By using this segmentation as attention, it is possible to estimate possible fiducial markers even if they are moving. A reinforcement learning algorithm is used to track the location and select an alternative closest fiducial point or fiducial marker if a location is not provided or found for a particular virtual fiducial marker. This two-stage network is trained both in the factory and in the field, allowing for continuous in-situ learning with field data. This applies, for example, to the U-Net architecture with the keypoint detection neural network described above.

[0025] The neural network can also be trained to have variable behavior of the virtual fiducial markers. For example, in the case of physical fiducial markers, different fiducial markers can be seen at different times when the head is rotated. During training, different views of the subject can be constructed so that different virtual fiducial markers are visible at different viewing angles. In this case, the number of fiducial markers provided by the image marking neural network is variable.

[0026] In another embodiment, the set of virtual fiducial markers is a set of predefined virtual fiducial markers. For example, the virtual fiducial markers can have a predefined relationship to the location of the subject's surface anatomical tissue, rather than a set of point clouds. The image marking neural network can be trained by providing images of the subject that have been manually labeled with the set of predefined virtual fiducial markers by a medical technician or physician. This works for structures such as the head, but also for other anatomical structures, such as the subject's chest region.

[0027] In another embodiment, execution of the machine executable instructions further causes the computing system to receive an initial medical image prior to commencing acquisition of medical imaging data. The initial medical image depicts an anatomical region of the subject. This may be, for example, a predefined anatomical region of the subject or adjacent or within the subject. Execution of the machine executable instructions further causes the computing system to receive a selected field of view identified in the initial medical image. The selected field of view may be, for example, an area desired to be imaged by a physician. Execution of the machine executable instructions further causes the computing system to calculate a registration between the image system coordinates of the virtual fiducial markers and the initial medical image. This then provides a reference between the imaging system coordinates and the initial medical image. This is used, for example, as a means to image the selected field of view.

[0028] Execution of the machine executable instructions further causes the computing system to configure pulse sequence commands to acquire medical imaging data from a selected field of view using the registration. This embodiment is beneficial because the area to be imaged by the subject is determined by using virtual fiducial markers. This allows data to be acquired without the use of a prior scout scan. This provides a reduced burden on the medical system to operate correctly to acquire initial medical images, such as magnetic resonance images.

[0029] In another embodiment, the initial medical image is an anatomical atlas image. This embodiment is beneficial because, for example, a physician can sit in his or her examination room and prepare the target area that is desired to be imaged to perform a diagnosis on the subject. The physician can do this using the anatomical atlas image, which is then translated using registration to properly perform the imaging.

[0030] In another embodiment, the initial medical image is a scout scan of the subject. This embodiment is beneficial because it automatically configures the medical system to use the scout scan to acquire medical imaging data.

[0031] In another embodiment, the medical imaging system is a magnetic resonance imaging system and the initial medical image is not a scout scan, in which case the medical system is implemented as a markerless and scoutless magnetic resonance imaging system.

[0032] In another embodiment, the initial medical image is a previous medical image of the subject. For example, even if a different imaging modality was used in the previous medical examination, the physician can indicate areas where it is desirable to perform more or a different imaging protocol. This previous medical image, even if it is an image of a different imaging modality, is registered to the fiducial markers to allow proper imaging of the selected field of view.

[0033] In another embodiment, execution of the machine executable instructions further causes the computing system to iteratively adjust the medical imaging system commands to acquire medical imaging data from the selected field of view using the registration in response to changes in the imaging system coordinates of the virtual reference markers. Changes in the coordinates of the virtual reference markers indicate that the subject has moved. By registering it to the imaging system coordinates, it infers how the selected field of view should be adjusted so that the subject's movement can be compensated for. This is beneficial as it reduces the effect of subject movement on the imaging process.

[0034] In another embodiment, the medical imaging system commands are adjusted to match the selected field of view with the latest imaging system coordinates of the virtual fiducial markers. For example, if the subject moves, the position of the selected field of view can be modified to match the latest coordinates. This has the effect of reducing the effects of motion on the imaging process.

[0035] In another embodiment, the pulse sequence commands are adjusted to match the selected field of view with the predicted coordinates of the imaging system coordinates of the virtual fiducial marker, which are determined using the velocity of the imaging system coordinates of the virtual fiducial marker. For example, the position of the virtual fiducial marker may be tracked over a period of time. Then, at a particular time, the velocity of the virtual fiducial marker may be used to predict the future position of the virtual fiducial marker. This has the effect of further mitigating the effects of motion.

[0036] In another embodiment, the medical system further comprises a subject-mounted gyroscope configured to provide gyroscope data representative of the subject's motion. Execution of the machine-executable instructions further causes the computing system to repeatedly receive gyroscope data from the subject-mounted gyroscope. Execution of the machine-executable instructions further causes the computing system to repeatedly determine the acceleration of the subject from the gyroscope data. Execution of the machine-executable instructions further causes the computing system to repeatedly calculate a predicted virtual reference marker velocity using the acceleration of the subject. Execution of the machine-executable instructions further causes the computing system to repeatedly calculate a predicted virtual reference marker position using the latest imaging system coordinates of the virtual reference marker and the predicted virtual reference marker velocity. This predicted reference marker velocity and position are then used to correct the position of the selected field of view based on this estimated or predicted reference marker position. This has the effect of reducing the impact of subject motion on image quality.

[0037] In another embodiment, the medical system further comprises a display. Execution of the machine executable instructions further causes the computing system to receive camera system coordinates of the set of virtual reference markers at the start of acquisition of the medical imaging data. Execution of the machine executable instructions further causes the computing system to calculate a position of an initial subject position indicator using the camera system coordinates of the set of virtual reference markers at the start of acquisition. Execution of the machine executable instructions further causes the computing system to persistently render the initial subject position indicator on the display. Execution of the machine executable instructions further causes the computing system to repeatedly calculate a position of a current position indicator using the camera system coordinates of the set of virtual reference markers.

[0038] One of the shortcomings of the system described in Slipsager et al. is that it does not utilize the system to inform the patient of patient position corrections even before the scan is completed, and it relies heavily on rigid point cloud templates that are prone to many errors. This embodiment overcomes this shortcoming by providing the above-mentioned indication.

[0039] Execution of the machine executable instructions further causes the computing system to repeatedly render the current subject position indicator on the display. This embodiment is advantageous because it provides a very good visual cue or tool for monitoring how the subject's position changes over the course of the examination. The initial subject position indicator indicates the subject's initial position, and then the current subject position indicator represents the subject's current position. This can be used in several different situations. A display can be provided to the subject so that the subject can see how he or she has moved. The display of both the current subject position indicator and the initial subject position indicator is also an aid to the subject in positioning himself or herself back to the initial position. This display can also be provided to the operator during operation of the medical system as an aid to the operator in understanding how much the subject has or has not moved during the examination.

[0040] In another embodiment, the initial subject position indicator is a rendering of a set of virtual reference markers positioned using the output camera system coordinates superimposed on the subject image at the start of acquisition, and the current subject position indicator is a rendering of a set (current or latest) of virtual reference markers positioned using the camera system coordinates superimposed on the current or latest subject image.

[0041] In another embodiment, the initial subject position indicator is a first object positioned using a combination of a set of virtual reference markers and camera system coordinates at the start of acquisition, and the current subject position indicator is a second object positioned using a combination of a set of virtual reference markers and camera system coordinates. In this case, a wide variety of objects can be positioned. For example, this does not have to be an actual image of the subject and virtual reference markers. In this case, this is, for example, a cartoon image roughly representing the subject. In other cases, the indicator is an object that is rotated and / or translated.

[0042] In another embodiment, the display shows an overlay of an initial medical or clinical image with the virtual fiducial markers, the initial medical or clinical image displaying, for example, an anatomical anomaly or abnormality, the overlay display helping the subject or operator to better understand the location and extent of the abnormality, if present.

[0043] In another embodiment, the medical imaging data is acquired in portions. Execution of the machine executable instructions further causes the computing system to correct each portion of the medical imaging data using the camera system coordinates of the virtual reference markers at the time each portion of the medical imaging data was acquired. This embodiment is beneficial because it provides a record of each individual portion of the medical imaging data with respect to the subject's position and / or movement. This can be used to retroactively correct the medical imaging data.

[0044] In another embodiment, the correction to the position of the medical imaging data is corrected using one of the following methods: by performing a rigid body rotation and / or translation, and by using a medical imaging data correction neural network configured to output corrected medical imaging data in response to receiving one of the output coordinates of the virtual reference marker at the time when each of the medical imaging data portions and medical imaging data positions was acquired. In these two cases, one is performed by correcting the raw medical imaging data using a translation. A neural network is also used for this purpose. The medical imaging data correction neural network is trained by taking the correct medical imaging data, dividing it into portions, and simulating the effect of the translation. For example, translating and / or rotating the measurements in k-space. Using the training reference data, the correct image or an image with the correct k-space data is corrected, and then the artificially translated data is used for training.

[0045] In another embodiment, execution of the machine-executable instructions further causes the computing system to reconstruct a clinical medical image from the medical imaging data.

[0046] In another embodiment, the medical system further comprises a radiation therapy system configured to irradiate the treatment zone. The treatment zone is within the imaging zone. Execution of the machine executable instructions further causes the computing system to receive radiation therapy control commands configured to control the radiation therapy system to irradiate the treatment zone. Execution of the machine executable instructions further causes the computing system to register the imaging system coordinates of the virtual fiducial marker to the clinical image. Execution of the machine executable instructions further causes the computing system to receive a position of the treatment zone in the clinical image. Execution of the machine executable instructions further causes the computing system to modify the radiation therapy control commands using the position of the treatment zone in the clinical image and the registration of the imaging system coordinates of the virtual fiducial marker to the clinical image. This embodiment is beneficial because it provides a more accurate use of the radiation therapy system. This embodiment has the advantage of reducing the setup time of the radiation treatment plan. There is a possibility that CT and / or MRI scans are not required for patient alignment because the alignment is performed using 3D real-time camera data and previous CT and / or MRI scans. A previous CT and / or MRI scan may be used as the initial medical image.

[0047] In another embodiment, the medical imaging system is a magnetic resonance imaging system.

[0048] In another embodiment, the medical imaging system is a combination of a positron emission tomography system and a computed tomography system.

[0049] In another embodiment, the medical imaging system is a positron emission tomography system.

[0050] In another embodiment, the medical imaging system is a single photon emission computed tomography system.

[0051] In another embodiment, the medical imaging system is a combination of a positron emission tomography system and a computed tomography system.

[0052] In another embodiment, the medical imaging system is a combination magnetic resonance imaging system and a positron emission tomography system.

[0053] In another aspect, the invention provides a method of operating a medical system, the medical system being configured to acquire medical imaging data depicting a subject from an imaging zone, the medical imaging system having an image system coordinate system, the medical system further comprising a camera system configured to acquire subject images of the subject during acquisition of the medical imaging data, the camera system having a camera system coordinate system, the method comprising acquiring the medical imaging data by controlling the medical imaging system with medical imaging system commands.

[0054] The method further comprises repeatedly controlling the camera system to acquire subject images during acquisition of medical imaging data. The method further comprises repeatedly receiving camera system coordinates of the virtual reference markers by inputting the subject images into an image marking neural network. The image marking neural network is configured to receive as an input an input image depicting a predetermined anatomical region of the subject. The image marking neural network is further configured to output camera system coordinates of the set of virtual reference markers in the input image in response to receiving the input image. The method comprises repeatedly providing imaging system coordinates of the virtual reference markers by repeatedly transforming the camera system coordinates of the set of virtual reference markers into the provided image system coordinates of the virtual reference markers using a coordinate system mapping. The coordinate system mapping is a mapping between a system coordinate system and a camera coordinate system.

[0055] In another aspect, the present invention provides a computer program comprising machine executable instructions executed by a computing system for controlling a medical system. The medical system comprises a medical imaging system configured to acquire medical imaging data depicting a subject from an imaging zone. The medical imaging system has an image system coordinate system. The medical system further comprises a camera system configured to acquire subject images of the subject during acquisition of the medical imaging data. The camera system has a camera coordinate system. The computer program also includes an image marking neural network, medical imaging system commands, and / or coordinate system mapping.

[0056] Execution of the machine-executable instructions causes the computing system to acquire medical imaging data by controlling the medical imaging system with medical imaging system commands. Execution of the machine-executable instructions further causes the computing system to repeatedly control the camera system to acquire subject images during acquisition of the medical imaging data. Execution of the machine-executable instructions further causes the computing system to repeatedly receive camera system coordinates of the virtual fiducial markers by inputting the subject images to an image marking neural network. The image marking neural network is configured to receive an input image depicting a predetermined anatomical region of the subject. The image marking neural network is further configured to output camera system coordinates of the virtual fiducial markers in the input image in response to receiving the input image.

[0057] Execution of the machine-executable instructions further causes the computing system to iteratively provide imaging system coordinates of the virtual reference markers by iteratively transforming the camera system coordinates of the set of virtual reference markers into the imaging system coordinates in which the virtual reference markers are provided using a coordinate system mapping, which is a map between the imaging system coordinate system and the camera coordinate system.

[0058] It should be understood that one or more of the above-described embodiments of the present invention may be combined, unless the combined embodiments are mutually exclusive.

[0059] As will be apparent to one skilled in the art, aspects of the invention may be embodied as an apparatus, method, or computer program product. Accordingly, aspects of the invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all generally referred to herein as a "circuit," "module," or "system." Additionally, aspects of the invention may take the form of a computer program product embodied in one or more computer-readable medium(s) having computer-executable code embodied therein.

[0060] Any combination of one or more computer readable media may be utilized. A computer readable medium may be a computer readable signal medium or a computer readable storage medium. As used herein, a "computer readable storage medium" encompasses any tangible storage medium that stores instructions executable by a processor or a computing system of a computing device. A computer readable storage medium may be referred to as a non-transitory computer readable storage medium. A computer readable storage medium may also be referred to as a tangible computer readable medium. In some embodiments, a computer readable storage medium may also store data that can be accessed by a computing system of a computing device. Examples of computer readable storage media include, but are not limited to, floppy disks, magnetic hard disk drives, solid state hard disks, flash memory, USB thumb drives, random access memory (RAM), read only memory (ROM), optical disks, magneto-optical disks, and register files of a computing system. Examples of optical disks include compact disks (CDs) and digital versatile disks (DVDs), such as CD-ROM, CD-RW, CD-R, DVD-ROM, DVD-RW, or DVD-R disks. The term computer-readable storage medium also refers to various types of recording media that can be accessed by a computer device over a network or communication link. For example, data may be retrieved via modem, over the Internet, or over a local area network. Computer executable code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

[0061] A computer-readable signal medium includes a propagated data signal having computer-executable code embodied therein, for example in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium is not a computer-readable storage medium, but is any computer-readable medium that can communicate, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device.

[0062] "Computer memory" or "memory" is an example of a computer-readable storage medium. Computer memory is any memory directly accessible to a computing system. "Computer storage" or "storage" is a further example of a computer-readable storage medium. Computer storage is any non-volatile computer-readable storage medium. In some embodiments, computer storage is also computer memory, or vice versa.

[0063] As used herein, a "computing system" encompasses electronic components capable of executing programs, machine-executable instructions, or computer-executable code. References to a computing system, including examples of a "computing system," should be interpreted as potentially including more than one computing system or processing core. A computing system is, for example, a multi-core processor. A computing system also refers to a collection of computing systems within a single computer system or a collection of computing systems distributed among multiple computer systems. The term computing system should be interpreted as potentially referring to a collection or network of computing devices, each of which comprises a processor or computing system. Machine-executable code or instructions are executed by multiple computing systems or processors within the same computing device or even distributed across multiple computing devices.

[0064] Machine executable instructions or computer executable code include instructions or programs that cause a processor or other computing system to perform aspects of the present invention. Computer executable code for performing operations for aspects of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Python, Smalltalk, C++, and traditional procedural programming languages ​​such as the "C" programming language or similar programming languages, and compiled into machine executable instructions. In some cases, the computer executable code is in the form of a high-level language or in a precompiled form, and is used in conjunction with an interpreter that generates the machine executable instructions on the fly. In other cases, the machine executable instructions or computer executable code is in the form of programming for programmable logic gate arrays.

[0065] The computer executable code may run entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet Service Provider).

[0066] Aspects of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block or part of the blocks of the flowcharts, diagrams, and / or block diagrams may be implemented by computer program instructions in the form of computer executable code, where applicable. Furthermore, it should be understood that combinations of blocks in different flowcharts, diagrams, and / or block diagrams may be combined, if not mutually exclusive. These computer program instructions may be provided to a general purpose computer, special purpose computer, or other programmable data processing device computing system to create a machine, such that the instructions, executed via a computer or other programmable data processing device computing system, create means for performing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0067] These machine-executable instructions or computer program instructions may also be stored on a computer-readable medium that can direct a computer, other programmable data processing apparatus, or other device to function in a particular manner, such as to create an article of manufacture that includes instructions that implement the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams, where the instructions stored on the computer-readable medium.

[0068] The machine-executable instructions or computer program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to perform a series of operational steps, such that the instructions, which execute on the computer or other programmable apparatus, create a computer-implemented process that provides a process for performing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0069] A "user interface" as used herein is an interface that allows a user or operator to interact with a computer or computer system. A "user interface" is also referred to as a "human interface device." A user interface provides information or data to an operator and / or receives information or data from an operator. A user interface allows a computer to receive input from an operator and provides output from the computer to a user. In other words, a user interface allows an operator to control or manipulate a computer, and the interface allows the computer to show the effect of the operator's control or manipulation. Displaying data or information on a display or graphical user interface is an example of providing information to an operator. A keyboard, a mouse, a trackball, a touchpad, a pointing stick, a graphics tablet, a joystick, a gamepad, a webcam, a headset, pedals, wired gloves, a remote control, and receiving data through an accelerometer are all examples of user interface components that allow for receiving information or data from an operator.

[0070] As used herein, a "hardware interface" encompasses an interface that allows a computing system of a computer system to interact with and / or control external computing devices and / or devices. A hardware interface allows a computing system to send control signals or instructions to external computing devices and / or devices. A hardware interface also allows a computing system to exchange data with external computing devices and / or devices. Examples of hardware interfaces include, but are not limited to, a universal serial bus, an IEEE 1394 port, a parallel port, an IEEE 1284 port, a serial port, an RS-232 port, an IEEE-488 port, a Bluetooth connection, a wireless local area network connection, a TCP / IP connection, an Ethernet connection, a control voltage interface, a MIDI interface, an analog input interface, and a digital input interface.

[0071] As used herein, a "display" or "display device" encompasses an output device or user interface adapted to display images or data. Displays output visual, audio, and tactile data. Examples of displays include, but are not limited to, computer monitors, television screens, touch screens, tactile electronic displays, Braille screens, cathode ray tubes (CRTs), storage tubes, bi-stable displays, electronic paper, vector displays, flat panel displays, vacuum fluorescent displays (VFs), light emitting diode (LED) displays, electroluminescent displays (ELDs), plasma display panels (PDPs), liquid crystal displays (LCDs), organic light emitting diode displays (OLEDs), projectors, and head mounted displays.

[0072] Medical imaging data is defined herein as the recorded measurements describing a subject provided by a medical imaging system. The medical imaging data is reconstructed into medical images, referred to as clinical medical images. A medical image is defined herein as a reconstructed two- or three-dimensional visualization of the anatomical data contained within the medical imaging data. This visualization may be performed using a computer.

[0073] K-space data is defined herein as the recorded measurements of radio frequency signals emitted by atomic spins using the antenna of a magnetic resonance machine during a magnetic resonance imaging scan. K-space is an example of medical imaging data.

[0074] A magnetic resonance imaging (MRI) image or MR image is defined herein as a reconstructed two- or three-dimensional visualization of anatomical data contained within magnetic resonance imaging data, which visualization may be performed using a computer.

[0075] Preferred embodiments of the invention will now be described, by way of example only, with reference to the drawings in which: [Brief description of the drawings]

[0076] [Figure 1] FIG. 1 illustrates an example of a medical system. [Diagram 2] FIG. 2 illustrates a method of using the medical system of FIG. 1. [Diagram 3] FIG. 1 illustrates a further example of a medical system. [Figure 4] 1 illustrates an example of a display provided to a medical imaging system operator and / or subject during a medical imaging system acquisition. [Diagram 5] FIG. 2 illustrates a further example of a medical imaging system. [Figure 6] FIG. 1 illustrates a further example of a medical system. [Figure 7]FIG. 1 illustrates the registration of a virtual fiducial marker to an initial medical image. [Figure 8] FIG. 13 shows a flow chart illustrating a further method. [Figure 9] FIG. 13 shows a flow chart illustrating a further method. [Figure 10] FIG. 13 shows a flow chart illustrating a further method. [Figure 11] FIG. 13 shows a flow chart illustrating a further method. [Figure 12] FIG. 13 shows a flow chart illustrating a further method. [Figure 13] FIG. 1 illustrates a further example of a medical system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0077] Like numbered elements in the figures are equivalent elements or perform the same function. If functionally equivalent, an element that has already been described is not necessarily described in a subsequent figure.

[0078] FIG. 1 illustrates an example of a medical system 100. The medical system 100 is shown to include a medical imaging system 102 and a computer 104. The computer 104 is intended to represent one or more computers located or distributed at one or more locations. The medical imaging system 102 is intended to be representative of different types of medical imaging systems. The medical imaging system may be, for example, a magnetic resonance imaging system, a positron emission tomography system, a single photon emission tomography system, a combination of a positron emission tomography system and a computed tomography system, or a combination of a magnetic resonance imaging system and a positron emission tomography system. The medical imaging system 102 has an imaging zone 106 in which medical imaging data can be acquired to image a subject 110. The subject 110 is shown lying on a subject support 112 and is at least partially within the imaging zone 106. In this example, the medical imaging system 102 has a controllable field of view 108. A camera system 114 in the medical imaging system 102 images an anatomical region of a subject 110. In this example, the region is the chest region.

[0079] The computer 104 is shown as including a computing system 120. The computing system 120 is intended to represent one or more processing cores or computing systems located in one or more locations. For example, the computer may be integrated into a medical imaging system or may be located in a removable manner. For example, the computer may be implemented such that it can be integrated with an existing motion detection and correction system as a plug-and-play interface.

[0080] Computing system 120 is shown in communication with a hardware interface 122 , an optional user interface 124 , and a memory 126 .

[0081] The memory 126 is shown as including machine executable instructions 130 that are used by the computing system 120 to enable the computing system 120 to perform various control and data processing and image processing tasks. The memory 126 is further shown as including an image marking neural network 132 that receives a subject image 138 and outputs a set of coordinates for a set of virtual reference markers in the coordinates of the camera system 114.

[0082] The memory 126 is further shown as including a set of medical imaging system commands 134. The medical imaging system commands 134 are a set of commands used by the computing system 120 to control the medical imaging system 102 to acquire medical imaging data 136. When the medical imaging system 102 is a magnetic resonance imaging system, the medical imaging system commands 134 are pulse sequence commands and the medical imaging data 136 is k-space data.

[0083] The memory 126 is further shown to include a subject image 138 acquired by the camera system 114. The memory 126 is further shown to include a coordinate system mapping 140 capable of mapping coordinates in the camera coordinate system to coordinates in the coordinate system of the medical imaging system 102. The memory 126 is further shown to include a set of camera system coordinates 142 of a set of virtual reference markers received from the image marking neural network 132 in response to receiving the subject image 138 as an input. The memory 126 is further shown to include image system coordinates 144 of a set of virtual reference markers calculated by transforming the camera system coordinates 142 of the set of virtual reference markers using the coordinate system mapping 140. The memory 146 is further shown to include a clinical image 146 reconstructed from the medical imaging data 136. The system coordinates 144 of the set of virtual reference markers may be used, for example, to improve the quality of the clinical image 146 in a number of different ways. This is used to correct the acquisition of medical imaging data 136 by tracking the movements of the subject 110, and also to retroactively correct the clinical images 146 during reconstruction.

[0084] 2 shows a flow chart illustrating a method of operating the medical imaging system 100 of FIG. 1. First, in step 200, medical imaging data 136 is acquired by controlling the medical imaging system 102 with the medical imaging system commands 134. Then, in step 202, the computing system 120 repeatedly controls the camera system 114 to acquire a subject image 138 during acquisition of the medical imaging data. It is considered that the step 200 is performed at least simultaneously with the step 202. The step 200 is also performed simultaneously with the steps 204 and 206. In step 204, the camera system coordinates 142 of the set of virtual reference markers are repeatedly received by inputting the subject image 138 to the image marking neural network 132. Then, in step 206, the image system coordinates 144 of the set of virtual reference markers are repeatedly acquired by transforming the camera system coordinates 142 of the set of virtual reference markers into the image system coordinates 144 using the coordinate system mapping 140. The method then proceeds to step 207, which is a decision box, asking the question "Is acquisition finished?" If the answer is yes, the method proceeds to step 208, where the clinical image 146 is reconstructed. If the answer is no, the method returns again to step 202, where steps 202, 204, and 206 are repeated until acquisition is finished. The acquisition started in step 200 continues as long as steps 202, 204, and 206 are repeated.

[0085] FIG. 3 illustrates a further example of a medical system 300. In this example, the medical imaging system is a magnetic resonance imaging system 302. The magnetic resonance imaging system 302 comprises a magnet 304. The magnet 304 is a superconducting cylindrical magnet with a bore 306 passing through it. Different types of magnets can be used, for example both split cylindrical magnets and so-called open magnets. Split cylindrical magnets are similar to standard cylindrical magnets, except that the cryostat is split into two sections to allow access to equal sides of the magnet, such magnets are used, for example, in conjunction with charged particle radiation therapy. Open magnets have two magnet sections, one above the other, with a space large enough to accommodate the subject between them, the arrangement of the two sections being similar to that of a Helmholtz coil. Open magnets are popular because they are less restrictive to the subject. Inside the cryostat of the cylindrical magnet is a collection of superconducting coils.

[0086] Within the bore 306 of the cylindrical magnet 304 is an imaging zone 106 where the magnetic field is strong and uniform enough to perform magnetic resonance imaging. A field of view 108 is shown within the imaging zone 106 for which k-space data (medical imaging data 136) is acquired. A subject 110 is shown as being supported by a subject support 112 such that at least a portion of the subject 110 is within the imaging zone 106.

[0087] Also within the magnet bore 306 are a set of magnetic field gradient coils 310 used for preliminary magnetic resonance data acquisition for spatially encoding magnetic spins within the imaging zone 106 of the magnet 304. The magnetic field gradient coils 310 are connected to a magnetic field gradient coil power supply 312. The magnetic field gradient coils 310 are intended to be representative. Typically, the magnetic field gradient coils 310 include three separate coil sets for spatially encoding in three orthogonal spatial directions. The magnetic field gradient power supply supplies current to the magnetic field gradient coils 310. The current supplied to the magnetic field gradient coils 310 is controlled as a function of time and may be ramped or pulsed.

[0088] Adjacent to the imaging zone 106 is a radio frequency coil 314 for manipulating the orientation of magnetic spins in the imaging zone 106 as well as receiving radio transmissions from the spins in the imaging zone 106. The radio frequency antenna includes multiple coil elements. The radio frequency antenna is also referred to as a channel or antenna. The radio frequency coil 314 is connected to a radio frequency transceiver 316. The radio frequency coil 314 and the radio frequency transceiver 316 are replaced by separate transmit and receive coils and separate transmitters and receivers. It should be understood that the radio frequency coil 314 and the radio frequency transceiver 316 are representative. The radio frequency coil 314 is also intended to represent a dedicated transmit antenna and a dedicated receive antenna. Similarly, the transceiver 316 also represents a separate transmitter and receiver. Additionally, the radio frequency coil 314 has multiple receive / transmit elements and the radio frequency transceiver 316 has multiple receive / transmit channels. For example, if a parallel imaging technique such as SENSE is performed, the radio frequency coil 314 has multiple coil elements.

[0089] The camera 114 images the subject 110's area of ​​interest. In this particular example, the subject's 110 chest region 322 is imaged. If another region is being imaged, such as the head or knees, the camera images these regions. The magnet bore 306 also has a display 330, which is optionally provided to the subject 110 to display initial and subsequent subject position indicators useful for the subject 110 to properly orient himself / herself after movement. The subject 110 also wears an optional subject-mounted gyroscope 332, which can track the subject's 110 movements and is useful for predicting the speed and therefore the path of the virtual reference markers.

[0090] The subject 110 is positioned within the bore of the magnet 106 such that the chest region of the subject 110 is located within the imaging zone 106. However, this is merely an example. Other regions of the subject 110, such as the subject's head, are also positioned to be imaged by the camera system 114 and the magnetic resonance imaging system 302.

[0091] The transceiver 316 , gradient controller 312 , camera system 114 , and display 330 are shown as connected to a hardware interface 1122 of the computer system 104 .

[0092] The medical system 300 is also useful for planning. The memory 126 is further shown as including an initial medical image 340 and a selection of a selected field of view 342. The initial medical image 340 is registered 344 to virtual fiducial markers, which are used to control the positioning of the field of view 108.

[0093] FIG. 4 illustrates one way of implementing the display 330. In this example, there is a first object 400 that represents the subject's position when the test begins. During the course of the test, the position of a virtual reference marker is tracked in the coordinate system of the camera 114. In FIG. 4, instead of displaying the initial and subsequent positions of the reference marker, the first object 400 and the second object 402 represent the starting and current positions of the reference marker. As the subject moves, the second object 402 rotates and translates. This is useful, for example, for a subject who wants to change his or her position. The first object 400 and the second object 402 are easier for the subject to understand in tight places. When the subject returns to the correct position, the second object 402 returns to the position of the first object 400.

[0094] FIG. 5 illustrates a further example of a medical system 500. The medical system 500 of FIG. 5 is similar to that shown in FIG. 1, except that it additionally comprises a radiation therapy system 502. The radiation therapy system 502 is configured to irradiate a treatment zone 504 and is controllable using radiation therapy control commands 506. The radiation therapy control commands 506 are, for example, registered (508) to the clinical image 146. The position of the treatment zone 504 is specified in the clinical medical image 146 and then used to modify the radiation therapy control commands 506 to better control the irradiation of the treatment zone 504. The medical system 500 includes the computer 104 and the camera 114 as shown in FIG. 1. The position of the treatment zone 504 and any 3D organ segmentation can be specified in the medical image in the same way that the FOV is specified in other embodiments.

[0095] Initial medical images, also referred to herein as scout images, are used to plan images for medical imaging system examinations such as MRI examinations. A scout is an image, in practice a scout is usually 3-5 low resolution images (2D scout) in 3 planes or a 3D low resolution scan, also used to cover the anatomy to be scanned. Scout images are used to determine the field of view for diagnostic sequences of medical devices such as PET, MR, CT, etc. Scout planning is an educated guess, and with a few scout images a reliable higher resolution image (clinical medical image) can be planned, and all subsequent images can be referenced in the same coordinate system.

[0096] Subject motion during magnetic resonance imaging (MRI) / PET / CT is an additional issue in obtaining good high-resolution scans, due to longer MRI scan times for some sequences and also due to patient pain leading to operator rescanning. The proposed solution proposes a markerless AI-based solution that automatically plans and tracks patient motions and automatically corrects for them. The proposed markerless solution also automatically derives markers (virtual reference markers) since the shape model-based landmarks of these external motion sensor inputs, e.g. 3DRGBD camera inputs, can be registered with the scout image landmarks that are automatically detected. Some examples also use a tracking and planning control system (TPCS) that uses the above inputs to retroactively perform minor organ motion corrections at run-time as well as predictively correct major bodily movements of the patient. The solution also analyzes various inputs like optics, pressure at run-time to estimate robust patient pose at run-time and provides feedback and indication to the patient during run-time. This artificial intelligence-based motion correction is also real-time, so it is fast and accurate as well as enhancing image quality.

[0097] Magnetic resonance imaging (MRI) and positron emission tomography (PET) are of great importance in the diagnosis and treatment of many neurological disorders. These modalities offer unique tissue contrast, but patient head motion is a significant problem due to the long image acquisition times. The degradation of image quality caused by patient motion can lead to a reduction in the detection of clinically relevant features, which can negatively impact diagnosis and treatment. It is estimated that patient motion adds $115,000 per scanner per year to the cost of MRI examinations. At present, there is no sign that the problem of subject motion during MRI examinations will be solved by hardware improvements. The possibility of accelerating imaging seems to be increasingly limited by biological constraints. Peripheral nerve stimulation limits gradient switching speeds, specific absorption rates (SAR) limit the use of RF excitation pulses, and T1 and T2 relaxation times constrain sequence repetition and echo times depending on the required contrast.

[0098] This problem is particularly severe in pediatric scans, where sedatives and anesthesia are frequently used, and can lead to side effects. To minimize such negative outcomes due to head motion, various motion correction (MC) methods for MRI and PET reconstruction have been proposed. For MRI, predictive MC, where the field of view (FOV) coordinate system is continuously updated during acquisition, has been demonstrated using various tracking techniques. Retrospective MRI MC uses motion information to retrospectively adjust the reconstruction to compensate for errors induced by motion. Unlike predictive MC, retrospective correction allows reconstruction with or without motion-corrected images. Only retrospective MC is possible for PET, since the acquisition cannot be dynamically adapted to compensate for motion. However, MC may be performed at various phases of PET reconstruction, from MC of raw list-mode data to MC of reconstructed image frames. These MC methods are generally based on the assumption that the exact head pose (position and orientation) is known during the scan.

[0099] Motion information can come from various sources, either directly from the acquired imaging data or using add-on motion tracking systems. Each approach has its own trade-offs in terms of accuracy, implementation complexity, and the need for additional hardware. Motion estimation from the imaging device itself does not require additional hardware, but it can make data acquisition and reconstruction more complex, while at the same time limiting the temporal and spatial resolution. In the context of MRI, motion data is often acquired by redundant sampling patterns that are either incorporated into the imaging acquisition or interleaved as "motion navigators". Meanwhile, various methods have been proposed to track markers attached to the subject. In the case of MRI, markers include field probes, active markers, gradient sensors, and optical targets.

[0100] Generally, markers (fiducial markers) are attached to the subject, and different attachment strategies have been presented for each of these markers to address this challenge. Stamping the patient's head has also been studied as a means to avoid the risk of marker detachment. However, feature extraction from stamps or facial characteristics alone is computationally expensive or unstable and has only been demonstrated for retrospective correction.

[0101] Data-driven motion detection in PET has shown promising results. However, it is difficult to distinguish between changes due to motion and functional changes in tracer distribution over time. These methods are analogous to the limited time resolution of motion estimation. In PET, optical marker tracking is somewhat simpler since the line of sight to the subject is not obstructed by the receiving coil as in MRI, allowing for more flexible marker design. Finally, simultaneous PET / MRI systems can also use inherent motion information in MRI data to estimate motion in both systems.

[0102] Until now, there have been no external motion tracking devices designed to work with both PET and MRI scanners. Existing solutions for MRI typically require attachment to the receive coil and do not consider the location of the PET detector. Conversely, motion trackers for PET scanners are not designed to work with the strong magnetic forces acting in the MRI environment.

[0103] In one example, the motion tracker is based on computer vision technology using a structured light surface scanner (camera system 114) that uses synchronized light modulators and cameras to continuously scan the patient's face or other anatomical regions. This approach does not require the attachment of optical markers, thus reducing clinical preparation time compared to marker-based solutions. Furthermore, no patient interaction is required, so there is no loss of patient comfort. Furthermore, this approach also eliminates tracking failures due to marker slippage. The system is capable of real patient motion tracking, and a tracking validity parameter (TVP) is used to ensure that the tracking is reliable and that inaccurate tracking is not used for motion compensation. Using inaccurate tracking for motion compensation would degrade the image instead of compensating it, which is unacceptable especially for clinical applications of predictive MC where image compensation is essential. To ensure tracking robustness, the TVP is calculated for each motion estimation and the estimation is accepted or rejected in real time.

[0104] Another example uses alternative marker selection (alternative virtual reference marker selection) built into the algorithm as explained above, leveraging reinforcement learning to constantly learn from the environment. This approach therefore eliminates the limitations caused by the Sager paper and reduces the need for TVP.

[0105] Head motion during PET, SPECT, and CT brain scans can cause artifacts and degrade image quality. Motion compensation can dramatically reduce such degradation, but motion-compensated brain imaging protocols are not used in routine clinical practice. This may be due to the lack of practical head tracking methods that can be easily integrated into busy clinical workflows. Although optical tracking provides highly accurate motion information, most optical systems are marker-based and require markers to be attached to the patient's head. Attached markers can be quite easily dislodged by rigid head movements while lying down, and more rigid fixation would require invasive fixation.

[0106] In one example, a markerless tracking system includes four CCD cameras arranged in pairs and pointed at either side of the face. During data acquisition, frames containing four synchronized images are collected continuously at 30 Hz. Unique features are detected for each frame and matched across images to determine 3D head landmarks. Once features are matched, the system builds a database of landmarks and their associated descriptors. This database steadily grows during the scan and is used by the tracking algorithm to estimate head pose changes.

[0107] To compare markerless tracking with a validated marker-based system, subjects wear a swimming cap or headband with large markers attached. To remove background features from areas such as the neck, clothing, and hair, various background masking techniques are used, including strip masking, where a basic mask is formed by rejecting fixed margins around the image edges, and face masking, which is determined using 16 facial landmarks.

[0108] When strip masking is used, 50-70 facial landmarks are used for pose calculation. The feature matching process was very reliable, with very few false matches recorded. Also, darker skin generally has lower contrast, so the system found fewer features, but it is still possible to track the movement.

[0109] The example differs from previous methods by using a combination of scout landmarks and motion sensor information (e.g., RGBD from a vision sensor) to plan an MR exam. It relies on tracking registered features, where an AI patient in the field uses a camera to learn the person's external features and uses this method to suggest their influence on the person's planned anatomical features. This is similar to image-to-anatomical tissue registration, but is much faster (real-time), and is used as a pre-estimate of the anatomical tissue given the camera image to reduce overall scan time, and also as a motion correction mechanism for the scan (e.g., MR) to provide feedback to both the patient and the technician. This also increases the accuracy of the markerless method by leveraging information provided by other sensors such as gyroscopes worn by the patient. Patients cannot easily understand the location of various anatomical tissues. Overlaying the MR scan with a 3D optical scan of the patient can promote good patient engagement.

[0110] 6 illustrates a further example of a medical system 600 comprising a magnetic resonance imaging system 302. The magnetic resonance imaging system comprises a gradient system 602, a pulse sequence server 604, a data acquisition server 606, an acquisition controller 608, and a patient positioning system 610. There is a vision probe 612 and a motion probe 614 that provide data to a tracking and planning control system 616. The tracking and planning control system 616 provides data to the acquisition controller 608 to coordinate the acquisition of magnetic resonance imaging data. The tracking and planning control system 616 is also used to control the gradient system for proper 3D imaging. The tracking and planning control system 616 also provides data to a display and feedback system 618 in the form of virtual marker feedback. The tracking and planning control system 616 also provides data to a reconstruction system 620 that reconstructs the magnetic resonance image. The display and feedback system 618 can, for example, control the display 330 or can also control an audio system 620 that provides instructions to the subject 110. The reconstruction system 620 also provides images to the display and feedback system 618 and data to a data server 622, which feeds the data to an anatomical atlas 624. The anatomical atlas 624 is used to provide a reference shape model to the tracking and planning control system 616.

[0111] 6 illustrates an MRI system 302 as the medical imaging system. However, this is merely exemplary. Other types of medical imaging systems may be substituted for the magnetic resonance imaging system 302. The entire medical system 600 is independent and can work with multi-modality and multi-vendor systems that include one or more of the following features: Tracking and planning control systems (multi-modality and multi-vendor systems) Tracking and Planning Control System 616: · 624 provides visual and motion input to the processor (computing system). The reinforcement learning system is initialized with an appropriate initial anatomical model and creates a mapping between visual and movement inputs and the initial anatomical tissue shape model. Provides output of FOV in magnet coordinate system. The output is fed to an acquisition controller 608, which predictively adjusts the patient position system 610, the gradient system 602, and retrospectively adjusts the reconstruction system 604. ·Reconstruction System 620 Retrospective motion correction Anatomical tissue model 624: To initialize the initial plan, a suitable anatomical model is collated using the patient data in the data server 622. ·Display & Feedback System 618

[0112] FIG. 7 shows several different images in four columns: a first column 700, a second column 702, a third column 704, and a fourth column 706. In the first column 700, a subject image 138 with a virtual fiducial marker 708 is shown. In the second column 702, an initial medical image 340 is displayed. These images also include a fiducial marker 708, and there is a registration 344 between the two. A selected field of view 710 is also marked on the initial medical image 340. The location of the fiducial marker in the images in column 700 can be used to position the selected field of view on the image in the third column 704. In this column, the selected field of view 710 is superimposed on the subject image 138. In column 706, there are several scout images 712 acquired for the corresponding field of view 710 in column 704.

[0113] FIG. 8 illustrates a method of using the medical system 600 depicted in FIG. 6. The method starts in block 800 and then proceeds to step 802 where a 3D optical image of the head is acquired. The method then uses a tracking and planning control system, block 616. The tracking and planning control system first executes planning and localization algorithms on the acquired optical image, step 804. Then, in step 806, the field of view is sent to the acquisition controller, which adjusts the gradient fields and starts the scan. The method then proceeds to step 808 where predictive motion correction using optical and / or gyroscope feedback is performed. The method then proceeds to decision box 810. "Is the scan complete?" is the question in box 810, and if the answer is "no", the method returns again to step 808 and this is repeated until the scan is completed. Once the scan is completed, the method proceeds to the use of the reconstruction system 620 and retrospective motion correction 812 is performed. After block 812, the method proceeds to the use of block 618 of FIG. 6 which is a display and feedback system. For example, a voice command or a visual command may be provided to the subject 110. The field of view from block 806 is also sent to a display in the feedback system 618. The method then ends at step 814.

[0114] FIG. 9 shows a flow chart showing in more detail the execution of the planning and localization algorithm on the acquired optical images, block 804. The method starts in step 900. The method then proceeds to step 902 and, optionally, simultaneously to step 904. In step 902, a 3D optical image region of the head is acquired. The method then proceeds to block 906. In optional step 904, a 3D scout image or a scout medical image is acquired. In step 906, using artificial intelligence, essential key points on the acquired images are identified. This is equivalent to identifying a set of virtual fiducial markers using the image marking neural network 132. The method then proceeds to step 908, where the field of view is estimated using the predicted key points or virtual fiducial markers as information. The method then proceeds to step 910, where the field of view parameters are sent to the scanner positioning system. The description then ends in block 912.

[0115] Figure 10 is a flow chart illustrating the details of block 806 of Figure 8 in greater detail. The method starts at block 1000 and proceeds to block 1002 where the current field of view parameters are input. This data is then provided to both block 1004 and block 1006. At block 1004, the optical system for tracking is initialized using the current field of view parameters. At block 1006, gyroscope data is acquired. Block 1004 proceeds to block 1008, which is a query box to determine if motion is detected. If motion is not detected, the method proceeds to execute boxes 1004 and 1006 again. If motion is detected using the gyroscope, the method proceeds to block 1012.

[0116] Block 1006 proceeds to block 1010, which is also a question box as to whether motion is detected using the gyroscope. If motion is not detected, the method returns again to steps 1004 and 1006. If motion is detected, the method proceeds to block 1012, similar to block 1008. In block 1012, the gyroscope data is used in conjunction with the optical data to estimate a new field of view. In block 1014, the current field of view is updated and the scanner position parameters are adjusted. After block 1014 is executed, the method then proceeds back to blocks 1004 and 1006.

[0117] FIG. 11 illustrates retrospective motion correction and describes block 812 of FIG. 8 in more detail. Initially, the method begins at block 1100. The method then proceeds to block 1102, which is an MRI image scan. In block 1104, the magnetic resonance image is converted to a k-space representation. In block 1106, a number n is randomly selected, a rigid body transformation is randomly applied n times to the magnetic resonance image scan and the optical scan, and the k-space corresponding to the time of occurrence of the transformation from the transformed image to the original k-space is plotted. The method then proceeds to block 1108, where a neural network is trained to reconstruct the magnetic resonance image scan using this corrected k-space. In block 1110, the trained neural network is used to reconstruct the image at run-time. The method then ends at block 1112. Thus, the method of FIG. 11 illustrates one way of training a neural network for use in retrospective motion correction.

[0118] FIG. 12 shows a flow chart illustrating block 618 of FIG. 6 in more detail. This is a description of the display and feedback system. The method illustrated in FIG. 12 starts at block 1200, and then continues with both block 1202 and block 1204. In block 1202, inputs from the scan results and the 3D optical image from the reconstruction system 620 are provided. Block 1202 then proceeds to block 1206. In this step, an overlay algorithm is performed to overlay the scan results on the optical image of the patient. Then, the block proceeds to step 1208. The overlaid image is sent to a display or printing device together with the original results. After block 1208, the method proceeds to block 1210, where the method ends. A parallel step 1204 is to provide inputs about the virtual marker positions and the motion detection results from the tracking and planning control system 616. After the end of block 1204, the method proceeds to block 1212, where an instruction is sent to the magnetic resonance audio system of the patient or subject 110 if significant motion is detected. The method then proceeds to block 1214 where the virtual markers overlaid on the optical scan are displayed to the radiologist, and optionally to the subject, and sent to any available display system for the patient.

[0119] 8 to 12 are again referred to below.

[0120] A. As shown in Figure 8, a markerless tracking, scoutless planning system is described below. Before the MR scan begins, a 3D optical image is taken and provided to the planning subsystem 804. The planning subsystem outputs the FOV parameters to the acquisition controller 608, and the magnetic field gradients are adjusted accordingly. Once the scan begins and motion occurs, the predictive motion compensation subsystem 808 sends feedback to the acquisition controller to adjust the FOV. This monitoring and feedback process continues until the scan is complete. Scan output is reconstructed using 812 The scan output is sent to a display system 618, which overlays the scan with a 3D optical image to make the output easier for the patient to understand.

[0121] B. As shown in FIG. 9, a planning system 804 is described below. a. The planning system receives (902) an acquired 3D optical image from the camera system and receives (904) a scout scan. b. Essential features are identified from the optical image and the scout scan, and the essential features are mutually registered (906). c. The FOV is estimated using the keypoints and this area is marked on the optical image to create virtual markers (908). d. The estimated FOV is sent to an acquisition controller system, which adjusts the magnetic field gradients (910).

[0122] C. A predictive motion compensation system 808 as shown in FIG. 1. The current FOV parameters are obtained as input from the acquisition controller 1002. 2. These parameters are used to initialize the optical and gyroscopic tracking systems (1004, 1006). 3. If either system detects motion, new FOV parameters are calculated using optical and gyroscope data, which updates the acquisition controller which adjusts the gradient magnetic fields. 4. The tracking system is then reinitialized according to the new FOV parameters. 5. This process continues until the scan is complete.

[0123] D. Retrospective Motion Correction System 812 a. Using an optical signal as an additional input b. Saves time by eliminating a post-reconstruction denoising step, as the algorithm denoises the scan during reconstruction i. The reconstruction algorithm is replaced by a neural network ii. The neural network is trained to be robust to k-space distortions induced by rigid body transformations iii. This robustness is introduced into the neural network by simulating rigid body transformations during the training process. The simulation is as follows: 1. The original image is converted to a k-space representation 2. A rigid body transformation is applied to the original image and the corresponding optical scan. 3. The transformed image is converted to a k-space representation. 4. After a certain number of slices, a new optical scan replaces the original optical scan, which indicates the patient's movement after a certain time t. 5. All k-space points sampled after time t are replaced with the k-space obtained from the transformed image. 6. Repeat steps 2-5 a random number of times (2-10). These indicate no moves. Each move is consecutive. iv. The neural network reconstructed images have similar orientation to the standard registered images.

[0124] E. As shown in FIG. 12, the Display & Feedback System 618 is described below. a. Display path (after scanning): A scan output is obtained from the i.MR system and a 3D optical image is obtained from the optical system (1202). ii. Overlaying the scanned anatomy with the 3D image using an overlay algorithm (1206) iii. This overlaid image and the scanned image are sent to a monitor or printing device for actual viewing / printing (1208) b. Patient Feedback: Using the MR device and audio system to the patient, virtual marker overlays are fed back to the patient along with instructions to help the patient maintain proper orientation during the scan (1204, 1212, 1214). Optionally, in the future, MR-compatible commercial patient devices such as Google Glass, Apple Air pods can be used.

[0125] 3. The following features are part of the algorithm: Optical scout registration scans for various modalities including MR, CT, PET: ·Find key points in the reference coordinate system from the MR-compatible conventional camera (3D), then predict the corresponding anatomical features in the images (similar to the MR images) and plan them in the conventional camera geometry. One of the points is the conversion between different coordinate systems. The camera coordinates, patient coordinates, and MRI coordinate systems are Patient motion estimation and compensation (both predictive and retrospective); Optical scout registration planning and are transformed into a single coordinate system that solves Include a gyroscope separately or combine it with the camera for accurate motion estimation MRI-compatible surgical planning - (registering a pre-performed MR scan to a live image from the camera for smoother and more precise surgery) · Smart planning on real images of different areas (both brain and other body parts) - Modeling covers clothing aspects too (IR camera can be used as well) - Combining IR camera with 3D camera. The camera can be positioned in various positions and can be fixed to the table or attached to the coil

[0126] Examples include one or more of the following features. 1. Optical scout: For registered planning and motion-compensated scanning, use cameras and gyroscopes to overlay MR scans and 3D optical scans for better understanding of the patient. Use scout images and registered optical scans for planning. 2. Create Virtual Markers for Tracking Based on Optical Images Uses scout images and registered optical scans for planning. Joint pose estimation algorithms allow creation of virtual markers. b. The created virtual markers are then tracked using an optical system by overlaying the created virtual markers with the optical scan using image processing techniques. c. The optical system for motion detection is further enhanced with information from the motion probe, which further improves accuracy. 3. Algorithm combining retrospective motion correction and reconstruction based on optical images To avoid excessive adjustments, the predictive motion correction is complemented by a retrospective motion correction system. The predictive motion correction system handles large deviations. When large deviations occur, the predictive motion correction system changes the positioning parameters of the gradient fields to realign them with the initial FOV, and the optical scan is readjusted accordingly. Smaller changes are handled by the retrospective motion correction system. b. A novel algorithm combining retrospective motion correction and reconstruction based on optical images helps to reduce the overall time and forms a suitable complement to markerless tracking and motion correction algorithms. 4. Overlay display & feedback system helps patient, operator and radiologist to better understand the location and extent of abnormalities if any. A user-friendly display / audio system (e.g. MR compatible devices like google glass / apple air pods) that helps general patient identify where the problem exists. 5. AI-based models can also be used to train data and map brain to anatomical locations, which can help create seamless studies, reduce overall scan times, and change the MRI scanning workflow in acquisitions. 6. Surgical planning can be significantly modified using 3D camera-based combined MRI scans of tumors or anomalies. To the best of our knowledge, this work has not been done in medical scanning, especially for scoutless surveillance planning.

[0127] F. Examples include one or more of the following features and / or advantages. Continuously learn both minor organ motion corrections and major patient physical movements at run time and apply them to perform corrections on scans. (Addressed by 616, 812) ·Does not use the power of patient-independent learning to accurately detect and track key points on planned subanatomical structures. (Addressed by 624) Using additional marker devices along with the sensors adds to the overall motion correction cost (addressed by 804) Patient discomfort caused by marker devices is eliminated through the use of the virtual markers of the present invention. (Addressed by 804) Predictive motion compensation avoids unnecessary compensation by limiting the use of predictive motion compensation to only compensating for large motions. Smaller motions are compensated for using a retrospective motion compensation system. (Addressed by 616 and 620) Motion compensation parameters and reconstruction can be combined to improve overall reconstruction time. (Addressed in 620, 812) Overlay display helps patients better understand the location and extent of abnormalities, if any, with patient guidance. (Addressed by 618)

[0128] G. Applications of the present invention include one or more of the following: Applications of the present invention are described in detail below. · Optical imaging is used together with scout images to perform FOV planning, thereby making the whole process more robust and facilitating subsequent motion tracking. Virtual markers can be used for subsequent optical tracking during scanning. · Use of body-worn gyroscopes to improve the performance of motion estimation in optical systems. Time savings due to improved retrospective motion correction based on optical scanning performed simultaneously with reconstruction. It provides predictive motion correction using camera coordinates. Overlay displays help the patient, operator and radiologist better understand the location and extent of abnormalities, if any.

[0129] Figure 13 illustrates a further example of a medical system 1300. The medical system 1300 of Figure 13 is similar to the medical system 100 of Figure 1 except for some configuration changes. The medical system 1300 is suitable for remote medical care. The medical imaging system 102 is controlled by a local controller 1302. The local controller is shown as being connected to the computer 104 via a cloud computing system 1304 (or network connection).

[0130] The computer 104 is shown as further comprising a user interface 1306. The initial medical image 340 is displayed and the operator can select the location of the field of view 108. Using the registration between the virtual fiducial markers and the initial medical image 304 or the registration between the virtual fiducial markers and the field of view 108, the acquisition of the medical imaging data 136 can be performed automatically. The memory 126 of the computer 104 is shown as including the same contents as the computer 104 of FIG. 1. However, the functionality of the computer 104 is transferred to the local controller 1302.

[0131] To assist in remote control of the medical imaging system 102, a display 330 is provided within the field of view of the subject 110. The display 330 is used by the subject to exercise self-control in positioning himself and limiting his movements during the examination. A display is also provided at the remote location to assist the operator, who then has knowledge of the subject's current position and whether the subject is moving during the examination.

[0132] While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive. The invention is not limited to the disclosed embodiments.

[0133] Other variations to the disclosed embodiments can be understood and realized by those skilled in the art, upon studying the drawings, the disclosure, and the appended claims, when practicing the claimed invention. In the claims, the word "comprises" does not exclude other elements or steps, and the singular elements do not exclude a plurality. A single processor or other unit fulfills the functions of several items recited in the claims. The mere fact that certain means are recited in mutually different dependent claims does not indicate that a combination of these means cannot be used to advantage. The computer program is stored / distributed on a suitable medium, such as an optical storage medium or a solid-state medium, supplied together with or as part of other hardware, but also distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. Any reference signs in the claims should not be interpreted as limiting the scope. [Explanation of symbols]

[0134] 100 Medical Systems 102 Medical Imaging Systems 104 Computer 106 Imaging Zone 108 Field of view 110 Target 112 Subject Support 114 Camera System 120 Computing Systems 122 Hardware Interface 124 User Interface 126 Memory 130 Machine Executable Instructions 132 Image Marking Neural Network 134 Medical Imaging System Commands 136 Medical Imaging Data 138 Subject Images 140 Coordinate System Mapping 142 Camera system coordinates of a set of virtual reference markers 144 Image system coordinates of a set of virtual fiducial markers 146 Clinical Images 200 Acquire medical imaging data by controlling a medical imaging system using medical imaging system commands 202 Repeatedly controlling a camera system to acquire subject images during acquisition of medical imaging data 204 By inputting the subject image into the image marking neural network, the camera system coordinates of the virtual reference marker are repeatedly received. 206 Iteratively provide imaging system coordinates of a set of virtual reference markers by iteratively transforming the camera system coordinates of the virtual reference markers into the provided imaging system coordinates of the virtual reference markers using coordinate system mapping. 207 Is the acquisition complete? 208 Reconstructing Clinical Images from Medical Imaging Data 300 Medical Systems 302 Magnetic Resonance Imaging System 304 Magnet 306 Magnet Bore 310 Magnetic field gradient coil 312 Magnetic field gradient coil power supply 314 Radio Frequency Coil 316 Transceiver 322 Chest area 330 Display 332 Subject-mounted gyroscope 340 Early Medical Images 342 Selected Field of View 344 Registration 400 First Object 402 Second Object 500 Medical Systems 502 Radiation Therapy System 504 Treatment Zone 506 Radiation Therapy Control Command 508 Registration to Clinical Images 600 Medical Systems 602 Gradient System 604 Pulse Sequence Server 606 Data Acquisition Server 608 Acquisition Controller 610 Patient Positioning System 612 Vision Probe 614 Motion Probe 616 Tracking and Planning Control System 700 1st row 702 Second Column 704 Third Column 706 Fourth Column 708 Virtual Reference Marker 710 Selected Field of View 712 Scout or initial medical images 800 start 802 Capture 3D optical images of the head 804 Execute planning and localization algorithms on acquired optical images 806 The field of view is sent to the acquisition controller, which adjusts the gradient fields and starts the scan. 808 Predictive Motion Compensation Using Optical and Gyroscopic Feedback 810 Is the scan complete? 812 Retrospective Motion Correction 814 End 900 start 902 Acquire a 3D optical image of the region of interest Get 904 3D scout images 906 Use AI to identify essential key points on captured images 908 Estimating Field of View Using Predicted Keypoint Information 910 Send field of view parameters to the scanner positioning system 912 End 1000 start 1002 Input: current field of view parameters 1004 Initialize the optical system for tracking using the current field of view parameters 1006 Get gyroscope data 1008 Motion Detected? 1010 Motion detected? 1012 Estimate new field of view using gyroscope and optical data 1014 Update the current field of view and adjust the scanner position parameters 1100 start 1102 MRI scan 1104 Converting MR images to k-space representation 1106 Randomly select a number n. Randomly apply the rigid body transformation to the MR image scan and the optical scan n times and plot the k-space versus time of occurrence of the transformation from the transformed image back to the original k-space. 1108 Using this corrected k-space, a neural network is trained to reconstruct the MR image scan. 1110 Use this trained neural network to reconstruct the image at runtime 1112 End 1200 start 1202 Input: Scanner results from 620 and 3D optical images 1204 Input: Virtual marker position and motion detection result from 616 1206 Run the overlay algorithm to overlay the scan results onto the optical image of the patient 1208 Send the overlaid image along with the original result to the display / print device 1210 End 1212 If significant movement is detected, instructions are sent to the patient's MR audio system. 1214 The virtual markers overlaid on the optical scan are displayed to the radiologist and sent to any available viewing system for the patient. 1300 Medical Systems 1302 Local Controller 1304 Cloud or Internet connection 1306 User Interface

Claims

1. A medical system comprising a medical imaging system, a camera system, a memory and a computing system, the medical imaging system having an imaging system coordinate system for acquiring medical imaging data depicting a subject from an imaging zone; the camera system having a camera coordinate system and acquiring subject images of the subject during acquisition of the medical imaging data; the memory storing machine executable instructions, medical imaging system commands, a coordinate system mapping, and an image marking neural network, the coordinate system mapping being a mapping between the imaging system coordinate system and the camera coordinate system, the image marking neural network receiving an input image depicting a predetermined anatomical region of the subject, the image marking neural network further responsive to receiving the input image outputting camera system coordinates of a set of virtual reference markers in the input image, and the medical imaging system commands controlling the medical imaging system to acquire the medical imaging data; The computing system controls the medical system, and execution of the machine-executable instructions causes the computing system to: acquiring the medical imaging data by controlling the medical imaging system using the medical imaging system commands; repeatedly controlling the camera system to acquire images of the subject during acquisition of the medical imaging data; repeatedly receiving the camera system coordinates of the virtual reference markers by inputting the subject image into the image marking neural network; iteratively providing the imaging system coordinates of the set of virtual reference markers by iteratively transforming the camera system coordinates of the virtual reference markers into the provided imaging system coordinates of the virtual reference markers using the coordinate system mapping; The medical system that makes it possible.

2. Execution of the machine-executable instructions further comprises: receiving an initial medical image prior to starting acquisition of said medical imaging data; receiving a selected field of view identified within the initial medical image; calculating a registration between the imaging system coordinates of the virtual fiducial marker and the initial medical image; constructing a pulse sequence command to acquire the medical imaging data from the selected field of view using the registration; The medical system of claim 1 .

3. The medical system of claim 2 , wherein the initial medical image is one of an anatomical atlas image, a scout scan of the subject, and a previous medical image of the subject.

4. The medical system of claim 2 or 3, wherein execution of the machine executable instructions further causes the computing system to iteratively adjust the medical imaging system commands in response to changes in the imaging system coordinates of the virtual reference marker to acquire the medical imaging data from a selected field of view using the registration.

5. the medical imaging system commands are adjusted to match the selected field of view with current imaging system coordinates of the virtual fiducial markers. the pulse sequence commands are adjusted to coincide with the selected field of view and a predicted coordinate in the imaging system coordinates of the virtual reference marker determined using a velocity in the imaging system coordinates of the virtual reference marker; and Combinations of these The medical system of claim 4 , wherein the medical system is any one of the following:

6. The medical system further comprises a subject-mounted gyroscope for providing gyroscope data describing a movement of the subject, and execution of the machine-executable instructions further comprises: repeatedly receiving the gyroscope data from the subject-mounted gyroscope; repeatedly determining an acceleration of the subject from the gyroscope data; iteratively calculating a predicted virtual reference marker velocity using the acceleration of the subject; iteratively calculating a predicted virtual reference marker position using current imaging system coordinates of the virtual reference marker and the predicted virtual reference marker velocity; The medical system according to claim 4 .

7. the medical system further comprises a display, and execution of the machine-executable instructions further comprises: receiving the camera system coordinates of the set of virtual reference markers at a start of the acquisition of the medical imaging data; calculating a position of an initial subject position indicator using the camera system coordinates of the set of virtual reference markers at the start of the acquisition; persistently rendering the initial subject location indicator on the display; and iteratively calculating a position of a current subject position indicator using the camera system coordinates of the set of virtual reference markers; repeatedly rendering said current subject location indicator on said display; The medical system according to claim 1 ,

8. the initial subject position indicator is a rendering of the set of virtual reference markers positioned using the output camera system coordinates superimposed on the subject image at the start of the acquisition, and the current subject position indicator is a rendering of the set of virtual reference markers positioned using the camera system coordinates superimposed on the subject image; and the initial subject position indicator is a first object positioned using a combination of the set of virtual reference markers in the camera system coordinates at the start of the acquisition, and the current subject position indicator is a second object positioned using a combination of the set of virtual reference markers in the camera system coordinates. The medical system of claim 7, wherein the medical system is any one of the following:

9. 9. The medical system of claim 1, wherein the medical imaging data is acquired in portions, and execution of the machine executable instructions further causes the computing system to correct each portion of the medical imaging data using the camera system coordinates of the virtual reference marker at the time each portion of the medical imaging data is acquired.

10. 10. The medical system of claim 9, wherein the correction to the portion of the medical imaging data is performed using one of the following methods: by performing a rigid body rotation and / or translation; and by using a medical image data correction neural network that outputs corrected medical imaging data in response to receiving one of the portion of the medical imaging data and the output coordinates of the virtual reference marker at the time each of the portions of the medical imaging data was acquired.

11. The medical system of claim 1 , wherein execution of the machine executable instructions further causes the computing system to reconstruct a clinical medical image from the medical imaging data.

12. The medical system further comprises a radiation therapy system for irradiating a treatment zone, the treatment zone being within the imaging zone, and execution of the machine-executable instructions further comprises: receiving radiation therapy control commands for controlling the radiation therapy system to irradiate the treatment zone; registering the imaging system coordinates of the virtual fiducial marker to the clinical image; receiving a location of the treatment zone within the clinical image; modifying the radiation therapy control commands using the location of the treatment zone in the clinical image and the registration of the imaging system coordinates of the virtual fiducial markers to the clinical image; The system of claim 11 .

13. 13. The medical system of claim 1, wherein the medical imaging data is one of a magnetic resonance imaging system, a computed tomography system, a positron emission tomography system, a single photon emission tomography system, a combination of a magnetic resonance imaging system and a positron emission tomography system, and a combination of a computed tomography system and a positron emission tomography system.

14. 11. A method of operating a medical system, the method comprising: a medical imaging system acquiring medical imaging data depicting a subject from an imaging zone, the medical imaging system having an imaging system coordinate system, the medical system comprising a camera system acquiring subject images of the subject during acquisition of the medical imaging data, the camera system having a camera coordinate system, the method comprising: acquiring the medical imaging data by controlling the medical imaging system using medical imaging system commands; repeatedly controlling a camera system to acquire images of the subject during acquisition of the medical imaging data; iteratively receiving camera system coordinates of the virtual fiducial markers by inputting images of the subject into an image marking neural network, the image marking neural network receiving an input image depicting a predetermined anatomical region of the subject, the image marking neural network further outputting camera system coordinates of a set of virtual fiducial markers in the input image in response to receiving the input image; iteratively providing imaging system coordinates of the set of virtual reference markers by iteratively transforming the camera system coordinates of the set of virtual reference markers into the provided imaging system coordinates of the virtual reference markers using a coordinate system mapping, the coordinate system mapping being a mapping between the imaging system coordinate system and the camera coordinate system; The method comprising:

15. 1. A computer program comprising machine executable instructions executed by a computing system to control a medical system, the medical system comprising a medical imaging system configured to acquire medical imaging data depicting a subject from an imaging zone, the medical imaging system having an imaging system coordinate system, the medical system comprising a camera system configured to acquire subject images of the subject during acquisition of the medical imaging data, the camera system having a camera coordinate system, and execution of the machine executable instructions causes the computing system to: acquiring the medical imaging data by controlling the medical imaging system using medical imaging system commands; repeatedly controlling the camera system to acquire images of the subject during acquisition of the medical imaging data; repeatedly receiving camera system coordinates of the virtual fiducial markers by inputting images of the subject into an image marking neural network, the image marking neural network receiving an input image depicting a predetermined anatomical region of the subject, the image marking neural network further outputting camera system coordinates of a set of virtual fiducial markers in the input image in response to receiving the input image; iteratively providing imaging system coordinates of the set of virtual reference markers by iteratively transforming the camera system coordinates of the set of virtual reference markers into the provided imaging system coordinates of the virtual reference markers using a coordinate system mapping, the coordinate system mapping being a mapping between the imaging system coordinate system and the camera coordinate system; A computer program that performs the following:

Images