Simulated 4D CT survey using 3D survey and RGB-D camera for improved scan planning

Abstract

A system FS and associated methods for facilitating the planning of raw data acquisition by a tomography imaging device IA are described. The system FS has an input interface IN capable of receiving an input 3D survey image V0 of a movable object OI occupying a first portion of 3D space, the survey image V0 being previously acquired by the tomography imaging device based on raw data acquired at a lower quality than intended for the planned raw data acquisition, and the survey image represents a first motion state of the object's motion. A prediction component PC predicts a predicted 3D survey image Vj, j>0, based on the input survey image V0, which can represent the object in other predicted motion states and the object in other spatial portions in those motion states. An output interface OUT provides the predicted survey image Vj to facilitate the planning of raw data acquisition.

Description

[Technical Field]

[0001] The present invention relates to a system, related methods, imaging configurations, computer program elements, and computer-readable media for facilitating the planning of raw data acquisition using a tomography imaging device. [Background technology]

[0002] CT (computed tomography) imaging typically begins with the acquisition of a survey image, also known as a scout or simply a "surview." Such survey images are acquired before the diagnostic scan. Such surveys may be used to assist the user in adjusting the field of view ("FOV") of the imaging device, such as a CT scanner. Such a field of view can be understood as a subset of the image region and is also called a "scan box" relative to a given target anatomical structure.

[0003] Compared to conventional 2D surveys (coronal and sagittal views), it is now possible to obtain 3D survey images with a similarly lower X-ray dose compared to the high doses expended on diagnostic scans. 3D survey images provide more detailed anatomical information than 2D projection images. This information can be used for manual or automated scan planning, such as selecting plan boxes and safety margins.

[0004] When returning to the 3D service, these are static images and do not provide temporal tissue motion caused, for example, by respiration or heart motion. Thus, the scan box plan is only valid for a specific respiratory state during the acquisition of projection data. For example, when a user plans the anatomical structure of the chest or heart, the true anatomical structure in some intermediate motion phases of the respiratory cycle may be unknown. To compensate for this, a default safety margin is added to the narrow scan box assuming that the organ of interest remains covered by a larger scan box even during changes caused by respiration. However, an incorrectly estimated margin may add insufficient image quality with unnecessary X-ray dose (the scan box is too large) or an incomplete anatomical view (the scan box is too small).

[0005] On the other hand, for workflow improvement in patient positioning, some vendors of tomographic modalities such as CT or MRI (magnetic resonance imager / imaging) provide automatic positioning based on RGB-D camera technology. The camera is typically mounted on the ceiling directly above the patient couch to monitor the position of the patient during scan preparation and its changes. In addition to the ceiling-mounted camera used for positioning, an in-gantry or in-bore camera can be used to monitor the patient during raw data acquisition.

SUMMARY OF THE INVENTION

PROBLEMS TO BE SOLVED BY THE INVENTION

[0006] Therefore, there may be a need for improved planning in tomographic raw data acquisition.

[0007] The object of the present invention is achieved by the subject matter of the independent claims into which further embodiments are incorporated in the dependent claims. It should be noted that the following aspects of the invention apply equally to the related methods, computer program elements, and computer-readable media.

MEANS FOR SOLVING THE PROBLEMS

[0008] According to a first aspect of the invention, a system is provided that facilitates the planning of raw data acquisition by a tomography imaging device, and this system is An input interface capable of receiving an input survey image (V0) of a movable object occupying a first portion of a 3D (three-dimensional) space, wherein the survey image is previously acquired by a tomography imaging device based on raw data acquired at a lower quality than intended for planned raw data acquisition, and the survey image represents a first motion state of the object's motion. A prediction component capable of predicting one or more predicted 3D survey images (Vj, j>0) that can represent one or more objects in one or more predicted motion states and one or more objects in one or more other parts of space that are in one or more predicted motion states, based on at least an input survey image, An output interface capable of providing at least one predicted survey image to facilitate the acquisition of the planned raw data, It holds.

[0009] The motion that an object of interest can perform should be interpreted broadly. Therefore, it may include one or more of the following: translation, rotation, and deformation (e.g., compression, contraction, expansion). Such motion may also include states of motion. In each or some of these states, the object may move to a different spatial position (relative to a reference position) and occupy a different portion of 3D space, at least partially.

[0010] In a medical setting, the object of interest may be at least a part of a target anatomical feature, such as at least a part of the heart of a mammal (human or animal), or at least a part of at least one of the lungs.

[0011] In one embodiment, the system includes a planner module that can calculate planning data based on one or more predicted 3D survey images.

[0012] In the embodiment, the planning data includes specifying a target portion of a 3D space that includes one or more other parts of the space, where the target portion of the space is a subset of the input survey images.

[0013] The aforementioned target portion of space (referred to as the "scanbox" in some imaging settings) includes / covers all voxel locations from which projection data is acquired.

[0014] The target portion is preferably determined as "tight," that is, the smallest portion of space that includes one or more other portions of space. Thus, the target portion is such that the moving object remains within its boundaries throughout its motion.

[0015] In one embodiment, the system includes a graphics display generator capable of generating a graphics display for display on a display device, the generated graphics display capable of visualizing one or more of the various motion states of an object based on one or more predicted 3D survey images.

[0016] The display device may be integrated into the operator console of the imaging equipment.

[0017] In the embodiment, the graphic display includes a video that can represent an animation of an object moving between different motion states, thereby occupying different parts of the space.

[0018] In one embodiment, the system has a user interface that allows the user to specify a target portion in space based on a graphical representation.

[0019] In embodiments, the prediction component can compute one or more predicted 3D survey images based on additional input data, including one or more of the following: i) video footage from which surrogate features of the object can be acquired by a non-ionizing radiation-based camera, or ii) a motion model for the motion of the object. The camera may be optical, IR (infrared), NIR (near-infrared), LIDAR (light detection and ranging), laser, etc. The camera may be a depth-sensing or ranging camera capable of recording a depth profile in which surrogate features change over time.

[0020] Surrogate features may be anatomically surrounding anatomical features, such as the skin of the chest. Generally, the subject may be an internal part of the patient (the anatomical feature of interest), while the surrogate features are external.

[0021] In this embodiment, the motion model is a machine learning model trained on training data, such as a convolutional artificial neural network.

[0022] The ML model may be a generative model, which may make it possible to predict a "virtual" survey without such a camera channel. However, non-generative ML, such as a regressor, or a supervised learning type, may also be used. Thus, two main embodiments may be assumed here: a one-channel input type (using an input survey) and a two-channel input type (using camera images of the input survey and the associated motion of surrogate features). The ML model may be used in the one-channel or two-channel embodiment.

[0023] For example, training data may include instances (samples) of historical or synthetically generated 3D survey images of the target example, and surface images based on non-ionizing radiation of each surrogate feature of the target in each example.

[0024] In addition or alternatively, training data may include sequences of such history or synthetic generative surveys from different patients representing a subject of interest in different motor states. The sequences can be split into subsequences and paired with another survey sample to obtain training pairs targeting each subsequence, with one such survey as the training input. Such pairs may be used to train regression models, particularly those configured for sequential data. Alternatively or in addition, generative models may be trained on such data.

[0025] In this embodiment, the motion model represents at least one motion component that can be extracted from the video footage, and the prediction component extrapolates a predicted 3D survey image by applying the motion component to the survey image.

[0026] Rigid or elastic registration may be used to spatially align the input survey with the camera image. This allows for the correlation of two movements: the movement of the object (target anatomical feature) by the survey and the movement of the surrogate feature by the camera image.

[0027] In embodiments, the system may include a transducer that can issue a warning signal if video footage from a non-ionizing radiation-based (e.g., depth-sensing) camera shows a deviation between the captured motion state and a model, and if the deviation violates predetermined tolerances.

[0028] In another embodiment, an imaging apparatus is provided that includes one or more systems according to the above claims, and one or more systems of a tomography imager, a camera, and a display device.

[0029] In yet another embodiment, a computer implementation method is provided that facilitates the planning of raw data acquisition by a tomography imaging device, and includes...

[0030] In yet another embodiment, a computer implementation method is provided for training a machine learning model used to plan the acquisition of raw data by a tomography imaging device based on training data, the model, when trained, can predict one or more such predicted images of a moving object in one or more different motion states, given a survey image of the moving object in a first motion state.

[0031] In yet another embodiment, a computer implementation is provided for acquiring training data to be used in a method for training a machine learning model to be used for planning raw data acquisition by a tomography imaging device, based on acquired training data, the model, once trained, can predict one or more such predicted survey images of an object in one or more different motion states, given a survey image of a moving object in a first motion state.

[0032] Training data may be obtained by searching for patients in a medical database such as PACS, which has records containing time-series video of at least one survey and surrogate feature. Alternatively, such survey-versus-surrogate feature video footage may be generated in a project for one or more hospitals with such cameras installed in the bore / gantry or reused to obtain such pairs of survey-versus-camera footage.

[0033] In yet another embodiment, a computer program element is provided which, when executed by at least one data processing system, is configured to cause that data processing system to perform any of the methods described above.

[0034] In yet another embodiment, at least one computer-readable medium is provided that stores program elements or models.

[0035] In some embodiments, the method proposed here involves combining information from a static 3D survey image with temporal motion information provided by a camera channel, such as the depth channel of a depth-sensing camera, which may or may not include an optical RGB channel. Such a camera may already be fitted for patient positioning or monitoring during scanning. It is proposed to use the images from such a camera for a new purpose, namely, for defining a scan box.

[0036] We propose using explicit or implicit motion models (such as respiratory motion models) to combine distance / range information from a presumably already installed camera with information from 3D survey images to generate an "animated" 4D survey, which is a prediction of how the original 3D survey image will change during a respiratory cycle. The original static survey image is then replaced by this animation, providing the user with additional valuable information for more accurate planning of diagnostic scans.

[0037] In some embodiments, a system is provided to facilitate the planning of projection data acquisition by a tomography imaging device, and the system is An input interface capable of receiving an input 3D survey image of an object occupying a first portion of 3D space, wherein the survey image is previously acquired based on projection data acquired at a dose lower than the high dose for planned projection data acquisition, and the survey image represents a first motion state of an object capable of ongoing motion. A prediction component that can predict plan data including a specified target portion in 3D space, where the imager acquires projection data at a higher dose, based on at least the input survey image, It holds.

[0038] The term "user" refers to a person who operates the imaging device or supervises the imaging procedure, such as a healthcare professional. In other words, the user is generally not the patient. However, in some autonomous imaging setups, the patient may perform certain functions that are normally performed by a healthcare professional or technician.

[0039] The terms “object of interest,” “target anatomical feature,” and “region of interest” are used interchangeably here. This refers to an actual part of the human body, or, in non-medical applications, the subject or purpose of imaging, particularly diagnostic scans / high-dose projection data acquisition. Features captured with a non-ionizing camera are not of diagnostic interest here. Such features (parts of anatomical structures, e.g., head, chest, torso, legs, etc.) are merely surrogates of the target anatomical feature of primary interest and purpose for the diagnostic scan. Such features may also be referred to here as surrogate features, such as the surrounding area (e.g., skin) of the anatomical feature or any other (external) anatomical structure that can be imaged with a non-ionizing camera. However, the motion state of the surrogate feature may be related to the motion state of the target anatomical feature.

[0040] A "motion state" concerns the transition of an object (such as a TAF or SF) from position to position in 3D space. Motion can include one, any two, or all three of the following: translation, rotation, or deformation. When an object like a TAF or SF moves, it occupies a different portion of 3D space, at least temporarily. Such 3D spaces are defined by 3D coordinates (voxels). Thus, different voxels within an image / inspection area are occupied in different motion states. The moment a particular portion of space is thus occupied is a "motion state." Therefore, each motion state has temporal and spatial components: the time the object occupies each portion of space and any space (a subset within 3D) that is thus occupied / filled. The space thus occupied can be identified by its respective voxel, i.e., the set of spatial coordinates of the point in the subspace thus occupied.

[0041] A “scanbox” is a 3D subset of the image region. It contains all the voxels from which projection data / tomographic raw data is collected. The term “box” as used in “scanbox” can actually refer to such a subset of rectangular shapes, but this is not mandatory; “scanbox” should be interpreted as any subset regardless of its shape. Preferably, the shape may be a function of the type of coordinate system used in the imaging geometry configuration or the image region. The scanbox, or the voxel positions it contains, may be defined as a subset of coordinates, as a geometric trajectory of equations, or by any form of parameterization that is convenient and appropriate for the imaging task at hand.

[0042] "Raw data," or tomography raw data, is a measurement result based on an interrogated signal interacting with patient tissue. Such signals may include X-rays in CT, RF signals in MRI, or gamma rays in PET. Such measurements are performed by detector devices (X-ray or gamma-ray sensing detectors, or TRF coils) in tomography imaging modalities such as CT, MRI, and nuclear imaging. Such detectors define a data space called the projection or raw data region, in contrast to another space, the image region, in 3D, which is composed of voxels (3D positions). These voxels are then supplemented by tomography algorithms (such as FBP) to form tomographic image values ​​that constitute a reconstructed cross-sectional image or a 3D volume. Projection data is an example of raw data acquired in a tomography setup, which is the primary assumption here. While the following primarily refers to such projection data, the principles described here, such as planning data / scanbox, etc., are equally applicable to other such tomography imaging setups.

[0043] A "survey" is a general 3D and reconstructed volume reconstructed from raw data, such as projections, using tomographic reconstruction algorithms. The image quality of such a survey is lower than that of the (target) volume reconstructed from higher IQ raw data. Such surveys have a lower IQ than diagnostic volumes, and some coarse details of anatomical structures (risk organs, abnormalities, and prominent landmarks) are still discernible, for example, in low-dose images. In a predicted virtual survey, when run as an animation, it may be possible to discern details of the movement of coarse anatomical structures, which is sufficient to plan the scanbox.

[0044] Generally, the term “machine learning” includes computerized configurations (or modules) that implement machine learning ("ML") algorithms. Some such ML algorithms work to tune a machine learning model configured to perform a task ("learn"). Other ML works directly on training data, in which case the training data may form (or part of) the model. This tuning or updating of the model is called “training.” Generally, task execution by an ML module can be measurably improved by training experience. Training experience may include appropriate training data and the model’s exposure to such data. The better the data represents the task to be learned, the better the task execution may be. Training experience helps improve performance when the training data well represents the distribution of examples in which the final system performance will be measured. See, for example, TM Mitchell, “Machine Learning”, page 2, section 1.1, page 6 1.2.1, McGraw-Hill, 1997. Performance may be measured by objective tests based on the output produced by the model in response to providing the model with test data. Performance may be defined in terms of a specific error rate that should be achieved for given test data.

[0045] Herein, exemplary embodiments of the present invention will be described with reference to the following drawings, which are not to scale unless otherwise specified. [Brief explanation of the drawing]

[0046] [Figure 1] This shows the imaging configuration, including the tomography imaging device. [Figure 2] This is a diagram illustrating the concept of a scanbox that can be used as planning data. [Figure 3] This shows a schematic block diagram of a facilitator system for predicting planning data, including scanboxes. [Figure 4] Figure 3 shows an embodiment of a facilitator system that uses explicit motion modeling. [Figure 5] Figure 3 is a block diagram of the training system for training the machine learning model used in implicit motion modeling in the embodiment of the facilitator system. [Figure 6] This flowchart shows a computer / computation implementation method that simplifies the specification of planned data for raw data acquisition using a tomography imaging device. [Figure 7] This flowchart shows a computer / computation implementation method for training machine learning models in implicit motion modeling for use in planning data specification. [Modes for carrying out the invention]

[0047] First, referring to Figure 1, a schematic block diagram of a medical imaging configuration (MAR) assumed in this embodiment is shown. The configuration MAR may include a medical imaging device (IA) (abbreviated as "imager"), preferably of the tomography X-ray-based type. Thus, the imager may be a computed tomography (CT) scanner, but intervention systems such as C-arm / U-arm types are not excluded. Other tomography modalities such as MRI, PET, etc. are also not excluded in other embodiments.

[0048] The configuration IAR further includes a computing system CS that is broadly operable to process data, including image data provided by the imager. The computing system may further enable control over the operation of the imager. The computing system may be located remotely from the imaging device IA, or it may be located close to it so as to be integrated into an operator console CS that allows a user to operate the imaging device and control imaging operations, particularly to acquire medical images for planning purposes such as diagnosis, treatment, and (radiotherapy, etc.).

[0049] More broadly, and as will be described more fully below, the computing system CS includes a facilitator system FS which may be used to facilitate planning data. Planning data may be used in a diagnostic scan to acquire section data or volume data V of the object of interest, such as a target anatomical feature TAF. Planning data may also include the definition of a scan box SB, a concept which will be discussed more deeply below in Figure 2. The scan box is data that enables the imager IA to be controlled to acquire appropriate diagnostic tomography raw data, such as projection data at a precise and appropriate dose for the imaging task at hand.

[0050] Broadly speaking, imaging may consist of two phases: a first planning (or preparation) phase and a subsequent second operation phase. The facilitator system FS is primarily operable in the planning phase to acquire scanbox data. The scanbox data may be used in the operation phase to control the imager IA to acquire diagnostic projection data λ relative to the target image V in the diagnostic scan.

[0051] The computing system CS, and in particular its facilitator system FS, may be implemented as a cloud solution running on one or more servers. The imaging device IA may be installed in a clinical facility such as a hospital. The computing system may be installed or used in a control room adjacent to the imaging room where the imager IA is located. In some embodiments, the computing system may be integrated with the imager IA. The imager IA may be communicably coupled to the computing system CS via a wired or wireless (or partially both) communication channel CC. The computing system CS may be configured as a fixed computing system such as a desktop computer, or as the aforementioned server, or as a mobile device such as a laptop computer, smartphone, or tablet. For example, the computing system CS may be configured on a workstation WS that can be associated with the imager IA.

[0052] Before going into more detail about the operation of the facilitator system FS, the components of the imager IA are first referenced to provide a more detailed explanation of the two-phase imaging process, which will help in the later explanation of the facilitator system FS.

[0053] The imaging device IA is operable to generate, in particular, acquire projection data λ, which is transmitted via a communication channel to the computer system CS for reconstruction into tomographic (cross-sectional) images. The computer system CS runs one or more reconstructor units, each implementing one or more reconstruction algorithms. Generally, the reconstruction algorithm implements mapping, which maps the projection data λ, located within the projection region, into the image region. The image region is part of 3D space and is located within the inspection region ER of the imaging device, while the projection region is 2D and is located within the (X-ray) detector XD of the imaging device IA.

[0054] The image region is conceptually composed of a 3D grid of 3D points (voxels), each point having its own 3D coordinate v(x,y,z) relative to a reference 3D coordinate system (X,Y,Z) with an origin that is defined and assumed to be given here. Such a reference coordinate system with such an origin is typically set when the imager IA is installed during the calibration procedure. Some of the voxel positions v are populated with values ​​by a reconstruction algorithm to construct a survey volume image V0, or in fact any other scan such as a subsequent diagnostic target volume V'.

[0055] As described above, the imaging device IA is preferably of the tomography type and preferably configured for multidirectional raw data acquisition such as projection image acquisition. Thus, the imaging device IA can acquire projection images λ along different projection directions α toward the examination area ER and therefore toward the target anatomical feature TAF within / in the patient PAT. Acquisition may, in embodiments, be performed by a rotating system in which at least the X-ray source XS is positioned in a movable gantry MG.

[0056] A movable gantry MG (and, in embodiments, together with the X-ray source XS) is rotatable within a fixed gantry SG around the examination area ER where the patient / ROI is located during imaging. Opposite the X-ray source of the movable gantry is an X-ray detector XD which can rotate with the gantry and X-ray source around the examination area ER to achieve different projection directions α.

[0057] As schematically shown in Figure 1, the patient's longitudinal axis or imaging axis Z may extend into the examination area ER during imaging. The patient PAT may lie on a patient support platform PS, such as a bed, which is positioned at least partially within the examination area ER during imaging. In some, but not all, embodiments, a helical imaging protocol is assumed in which there is relative lateral motion along the longitudinal axis Z between the X-ray source XS and the patient PAT. For example, the patient support platform PS may advance through the examination area ER during multi-directional projection image acquisition, for example, during the rotation of the X-ray source XS around the patient.

[0058] The CT scanner setup shown in Figure 1 is merely one embodiment, and other tomography imaging equipment such as C-arm or U-arm scanners, cone-beam CT configurations, and mammography imagers are not excluded here. In some embodiments, a C-arm cone-beam imager is preferred here. In addition, multi-directional acquisition capability is not necessarily obtained from a rotating system as shown in Figure 1. Non-rotating imaging systems, such as fourth- or fifth-generation CT scanners, in which multiple X-ray sources are arranged around the examination area, for example, in a source annular structure, are also conceivable. In addition or alternatively, the detector XD may be arranged around the examination area as a detector annular structure. Thus, in such a system, there is no rotation of the X-ray source XS or the detector XD, or both.

[0059] An operator console (OC) may exist in which a user, such as a medical professional, controls image manipulation. For example, the user may request the start of image acquisition, or request reconstruction or other operations, or start sending data to a computing system (CS), or stop such transmission as needed. The operator console (OC) may incorporate a display device that shows control parameters, etc., but in the console, the display device may be used by a facilitator system (FS), as will be described below.

[0060] During imaging, each X-ray beam XB is emitted along a different projection direction α from the focal spot of the X-ray light source XS. Each beam XB passes through the examination area where the patient is located. The X-ray radiation interacts with the patient's tissue. The X-ray beam XB is modified as a result of this interaction. Generally, such modification of the X-ray beam XB involves attenuation and scattering of the original incident X-ray beam. The modified X-ray radiation is then detected as a spatial distribution of varying intensities by the X-ray sensitive pixels of the detector XD.

[0061] Here, it is not necessary to acquire a projected image λ over the entire 360° angular range around the inspection area ER. Acquisition over a partial angular range, such as 270°, 180°, or less, may suffice. The X-ray detector is preferably configured for acquiring a 2D projected image with rows and columns of intensity values ​​aligned by detector pixels. That is, the detector pixels themselves may be arranged in a matrix layout. Such a 2D layout is used in divergent imaging geometric configurations such as cone or fan beam. However, one-dimensional detector pixel layouts (such as along a single line) are not excluded here, nor are parallel beam geometric configurations.

[0062] The reconstructor RECON implements one or more reconstruction algorithms for processing diagnostic projection images λ. Specifically, the reconstructor RECON may compute a tomographic target image V of the examination area (where the patient is located) for diagnostic, therapeutic, or other purposes. The reconstructor RECON may generate cross-sectional volume image data ("image volume") V. However, this does not preclude generating a single image slice within the examination area as needed. Thus, the reconstructed image, denoted as V, may include the entire volume, a partial volume, or a specific cross-section through it. Volume reconstruction may be facilitated by helical movement and / or 2D layout of the X-ray detector XD.

[0063] The scanbox SB (data) may be defined automatically or, in embodiments, by the user, particularly by using user input UI functions supported by the facilitator system FS. Broadly speaking, the scanbox CB is a portion of the 3D space within the §D image area. It is therefore a 3D object, but may be visualized as a rectangle in a 2D view, hence the name scanbox. The scanbox defines the volume to be scanned in order to obtain projection data from which the volume of space indicated by the scanbox is reconstructed.

[0064] The reconstructed volume image V may be stored in memory MEM or processed in other ways as needed. The reconstructed volume image V may be visualized by a visualizer VIZ. The visualizer VIZ may generate a graphic representation of the volume or a given slice. The graphic representation may be displayed on a display device DD. The visualizer VIZ may map a cross section of the image volume V, or the entire image volume V, to gray values ​​or a color palette. The visualizer VIZ may control a video circuit, via a suitable interface, to control the video circuit that performs the graphic representation on the display device DD. In addition to or instead of displaying in this way, the reconstructed image V may be stored in memory for later review or for other types of processing. Such memory may include an image repository such as a database (e.g., PACS) or other (preferably) non-volatile data storage device.

[0065] The reconstructed volume image V can be manipulated, for example, by reformatting, to define a cross-section different from that defined by the imaging geometric configuration. Such reformatting may allow medical users to better identify the tissue type or anatomical details within the patient, depending on medical purposes such as diagnosis or preparation for some type of treatment.

[0066] The reconstructed target volume V is obtained from the reconstruction of diagnostic projection data λ during the operation phase, based on information from scanbox SB. This reconstructed image V is intended to be a diagnostic image representing the target anatomical feature TAF, such as a target anatomical structure, organ, part of an organ, or group of organs, different tissue types, etc., with sufficient contrast to safely inform treatment or diagnostic decisions, or other medical decisions, which may involve further imaging sessions using other imaging modalities or other tasks (such as tests) that need to be performed based on the aforementioned image.

[0067] The (target) projection image λ from which the target volume V is reconstructed is acquired with a sufficient dose by appropriately controlling the dose used via the operator console OC. This can be done by controlling the voltage and / or amperage settings of the X-ray source XS tube to ensure a specific diagnostic image quality. Such a target projection image acquired with a sufficiently high dose may, in accordance with established terminology, also be referred to here as the diagnostic projection image λ. However, this naming convention does not prevent this projection data λ and its reconstruction V from being used for non-diagnostic tasks, such as treatment (e.g., in a catheterization lab), planning, or other tasks.

[0068] The target anatomical feature (TAF) is related to the medical purpose of imaging. Therefore, if it is necessary to examine the patient's liver, the target anatomical feature (TAF) is the liver, and the purpose and subject of the abdominal scan is the liver.

[0069] Each target anatomical feature (TAF) is generally associated with a set of specifications that define specific preferred imaging settings, required image contrast to be achieved, radiation dose to be used for the target anatomical feature TAF for a given purpose, voltage / ampere settings of the X source used, collimation, etc., depending on the imaging protocol, preferably the patient's biological characteristics (age, weight, sex, height, BMI, medical records, etc.). In short, the imaging protocol encapsulates medical knowledge regarding any given imaging task, purpose, target anatomical feature TAF, etc.

[0070] The facilitator system FS operates during the planning or preparation phase prior to the operational phase in which the target volume V is acquired.

[0071] In the planning phase, an initial survey image V0 is acquired. This initial image V0 is preferably a 3D (image region) survey volume, which is reconstructed by the reconstructor RECON (or another reconstructor) from a first (non-diagnostic) set of projected images λ'. However, this first / initial set of projected images λ0 is acquired in the planning phase, and later in the operation phase, in the first acquisition by the imager IA at a lower image quality than the projected images λ used to reconstruct the target image V. Thus, there are two acquisitions, one in the planning phase, which acquires low-dose non-diagnostic prediction data λ', and the later acquisition in the operation phase, which acquires high-dose diagnostic prediction data λ.

[0072] In particular, when acquiring λ0, a lower radiation dose is generated compared to the subsequent dose for the diagnostic scan to acquire the diagnostic projection image λ. This is to conserve the dose to the patient PAT, and also because the 3D survey volume V0 serves a completely different purpose than the target volume V, the purpose of which, as extended here, is essentially one of navigation, and the survey image V0 is used to define the scan box relative to the target anatomical feature TAF. The scan box thus controls the imager to acquire higher dose / higher-dose diagnostic projection data λ for a smaller field of view (FOV) by scan box SB, focused on the target anatomical feature TAF. Thus, in survey, a much wider FOV was used but the dose was lower; no planning was required there. However, when the facilitator FS defines the scan box, a diagnostic scan is performed in a smaller FOV but with a higher dose to acquire the target volume relative to the target anatomical feature TAF with a higher IQ diagnostic grade.

[0073] Referring more closely to the facilitator system FS, it is configured to facilitate the definition of scanbox SB, where diagnostic quality data λ is acquired in a new expected diagnostic acquisition by imager IA. Such diagnostic quality projection data λ is reconstructed by reconstructor RECON to target volume V at an image quality ("IQ") higher than the IQ of the low-dose projection data λ0 based on the survey image V0. In the operational phase, the new diagnostic acquisition by imager IA is expected to occur in the same laboratory, for the same patient, after the survey volume V0 has been acquired, as calculated by the facilitator. This is a preferred workflow. Thus, the patient preferably remains in approximately the same position within the imager bore BR for the survey scan and the subsequent diagnostic scan. However, theoretically, if necessary, it may be possible, for example, using appropriate registration techniques, to acquire a survey scan in a first imaging session on a given day, and for the patient to return for an imaging session on another day or later on the same day, and then acquire a diagnostic scan. However, such a fragmented workflow is not very desirable here, and as mentioned earlier, the option of the patient PAT remaining within the imager's bore BR for both scans is preferred.

[0074] The target volume V provides a diagnostic quality view of the target anatomical feature TAF, or any other object of interest OI. While the object of interest OI is primarily referred to here as the target anatomical feature TAF, the two terms are used interchangeably. The term "object of interest OI" as used herein may be understood to refer to foreign bodies (if present), such as implants, and may also be applicable to non-medical uses as described in these embodiments.

[0075] The following will primarily be described with reference to tomographic radiography, such as the donut-shaped X-ray scanner, or C-arm scanner, or any other design shown in Figure 1. However, the principles of scanbox discovery and definition assumed herein are not limited to X-ray-based imaging such as CT, but can also be usefully practiced in tomographic modalities such as MRI, PET / SPECT, or any other where planning data must be established.

[0076] Before going into more detail about the operation of the facilitator FS, or "scanbox standardizer," Figure 2, which outlines the concept of a scanbox as it may be conceived here, is referenced.

[0077] First, refer to Figure 2A, which shows a schematic diagram of the imaging area ER as defined by the bore BR of the imager IA. Since a CT imager is mainly assumed here, the imaging area ER has a cylindrical shape. The shape depends on the structure of the imaging device and therefore may not be cylindrical in other modalities. In MRI, the imaging area ER is also generally cylindrical. The imaging area ER can also be called the imaging area. As mentioned above, the image area is part of 3D space and conceptually consists of a grid of 3D positions (voxels).

[0078] A patient is typically a target anatomical feature (TAF) that is present in, or at least present in, the examination area. A scanbox is planning data that can be used for the diagnostic (e.g., high-dose) imaging of the target anatomical feature (TAF). A scanbox SB is a definition of a portion of 3D space. A scanbox is a 3D sub-part of the image area. A scanbox is a (mathematically true) sub-volume of the examination area (ER). Therefore, a scanbox SB is a (mathematically true) subset of the examination area. A scanbox is positioned and sized to include (cover) at least the target anatomical feature (TAF). In other words, a scanbox is a set of voxel locations in which at least sometimes a portion of the target anatomical feature (TAF) resides. A scanbox can be defined by any geometric shape. This does not necessarily have to be a box (cuboid) as shown in Figure 2, but may instead be defined as an ellipsoid, a sphere, or any other shape depending on the implementation. For example, the shape may depend particularly on the common image area coordinate system (X,Y,Z). When the image region coordinate system is a Cartesian coordinate system, the cuboid definition is preferred, as is common, due to its computational simplicity and the low latency performance of the facilitator system FS. However, a spherical definition or similar may be more appropriate when a polar coordinate system or the like is used.

[0079] The scanbox SB defines the set of voxel locations from which projection data will be collected during the next diagnostic projection data acquisition. These voxel locations form part of a 3D subset defined by the spatial constraints of the scanbox SB.

[0080] Once the scanbox is sufficiently defined, it can be passed to the control interface CL. The control interface CL converts the spatial information from the scanbox SB into imaging control commands or imaging parameters to control the imaging apparatus IA to achieve the acquisition of corresponding projection data for all voxels within the scanbox. Thus, the scanbox defines such imaging control commands or imaging parameters. Such control operations may also relate to the (relative) translation of the patient table PS and / or the rotation of the X-ray source XS around the imaging axis Z. Such control operations may determine the position and length of the cross-section (length) along the imaging axis Z from which the projection data is collected. The imaging control commands or imaging parameters may define other aspects of the imaging geometric configuration, such as the beam shape used (e.g., collimation), the starting point of the X-ray tube rotation, the pitch (ratio of translation along axis Z to the number of rotations around axis Z), and other imaging parameters or commands.

[0081] The scan box SB can be provided to the imaging control interface CL by the facilitator FS in one of several different expressions or specifications, depending on the available control circuit / code, and all are conceivable in different embodiments. Generally, as is evident from Figure 2A, the scan box is a specification of voxel positions. For example, the scan box SB may be provided as a set of all voxel positions that form the scan box. Alternatively, and preferably for low-latency performance, the scan box may be provided with appropriate parameterization. For example, the coordinates of the reference point of the scan box may be provided in addition to the size value. More specifically, for example, in the case of a rectangular scan box, the corner or center point may be provided, elongated by the length of the diagonal of the rectangular prism. If a ball shape is preferred, the radius / diameter and center point, etc., may be provided. Alternatively, the reference point (corner point) and the length, width, and height of the scan box are provided. Any other parameterization, specification of any kind is conceivable, as long as it is clearly defined for each voxel position within the image area / inspection area, whether or not it forms part of the scan box.

[0082] The size of the scan box should be such that it encompasses the entire target anatomical feature (TAF). However, if one or more of the target anatomical features, such as parts of the human lungs or heart, are subject to motion, care must be taken to appropriately define the size and position of the scan box to account for such motion. Therefore, the anatomical feature TAF may be subject to motions such as translation, rotation, and deformation, and different spatial portions within the examination area may be assumed for different motion states. This motion may or may not be periodic.

[0083] The precise definition of the scanbox is preferable here to ensure that all of the target feature TAF are always included in different motion states, but preferably not larger than necessary, as this may result in unnecessary X-ray dose exposure to patient tissue unrelated to the imaging task. Such strictness is a spatial minimization condition.

[0084] Figures 2B and 2C illustrate what may be problematic when defining the scan box SB of the target anatomical feature TAF that is the object of motion. Motion states are indicated by different line styles. Motion states indicated by dashed lines represent different motion states compared to motion states indicated by solid lines. Thus, the motion states represented in Figures 2B and 2C can be understood as changes between compression (Figure 2C) and expansion (Figure 2B), such as the changes in the human heart that transmits pumping function. As can also be seen in Figures 2B and 2C, when the target anatomical feature TAF transitions between different motion states, it occupies different portions of the surrounding space of the image region ER and occupies (or does not occupy) specific voxel locations. For clarity, the scan box SB in Figures 2B-D is shown along the Z direction, i.e., along only one spatial dimension.

[0085] In the example shown in Figure 2B, the scanbox SB is selected to be too small. During movement, for example, during extension, the target anatomical feature TAF extends beyond the limits of the scanbox. On the other hand, in the case shown in Figure 2C, the scanbox is defined too broadly, leaving an undesirable spatial clearance Δ because dose is spent on voxel points in clearances that do not provide spatial information about the target anatomical feature TAF.

[0086] Figure 2D shows a correctly selected scanbox following the principle of minimization or strictness described above. Therefore, the interface of scanbox SB is preferably selected to contact a boundary point on the target anatomical feature, particularly the boundary point on the target anatomical feature TAF that moves furthest as it transitions through the motion state. The "dead space" Δ of voxels that are not visited by the target anatomical feature TAF during motion is preferably approximately zero. Thus, there are virtually no points within scanbox SB that are never visited.

[0087] Therefore, the facilitator FS is configured to facilitate spatially defining the scanbox such that all points of the target anatomical feature TAF always remain within the limits of the thus defined scanbox SB, without any voxels being present that are not visited by the target anatomical feature TAF during transitions through different motion states. Thus, the scanbox itself remains stationary, but its size and position are defined such that the target anatomical feature TAF always remains within the box, and the "dead space" Δ is approximately zero, as shown in the strictly defined example of scanbox SB in Figure 2D.

[0088] More broadly, the facilitator system FS can assist the user in defining the voxel positions of the scan box. This can be done by defining the size and position of the bounding box in the image region ER. The facilitator achieves this by processing the acquired survey image V0 of the target anatomical feature TAF as input, in a predetermined ("initial" or "reference") motion state, i.e., in the initial position and range of the target anatomical feature TAF. Thus, the target anatomical feature TAF in this initial motion state occupies the initial portion of the 3D space of the examination region ER. The aforementioned furthest motion state of the target anatomical feature TAF is taken relative to this initial motion state recorded in the (initial) survey image V0. The survey image V0 is volume data that captures a reasonably large range of 3D examination area ER, either the entire examination area ER or a portion thereof, the portion of which includes the target anatomical feature TAF, but is a priori known to include substantial (non-zero anyway) dead space Δ in addition to other surrounding anatomical features, thus demonstrating survey anatomical information about how the target anatomical feature TAF is embedded in the overall / larger anatomical structure surrounding the patient.

[0089] The facilitator system FS processes this survey image V0 as input data and generates predictions of multiple "virtual" survey images Vj based on the survey image V0. Preferably, each virtual survey Vj represents a target anatomical feature TAF in a different predicted motion state, distinct from the initial motion state determined by V0. Hereinafter, for brevity, the real / initial and virtual survey images will be simply referred to as "surveys," and "volumes" or "images" will be omitted. One or more virtual survey Vj calculated by the facilitator FS preferably have the same size and dimensions as the initial survey. Thus, the virtual survey Vj is also a volume representing the 3D space within the examination area, capturing the image portion with dead space Δ>0. However, while the initial survey V0 is based on measured projection data, the virtual surveys are predictions and therefore not based on, or at least not directly based on, such measured prediction data. Therefore, no further dose is generated when generating the virtual survey Vj, j>0. The predicted survey image Vj can be conceptualized as an extrapolation from the initial input survey volume V0.

[0090] Preferably, the number of predicted virtual survey Vj for different time points (represented by index j) is such that they cover all motion states of the subject, including those furthest from a given reference motion state actually represented / captured in the reconstructed input survey V0. Theoretically, as is actually assumed in some embodiments, it may be sufficient to predict only one such virtual survey image Vj*. If such a single survey is predicted, it preferably represents the motion state (extreme motion state) of the anatomical feature that is spatially furthest from the initial motion state of the target anatomical feature captured in the initially measured survey V0. However, preferably, in some embodiments, more than one predicted survey image is calculated by the facilitator system to represent intermediate motion states between the initial state from the initial survey V0 and the extreme motion state predicted and represented in the extreme survey Vj*. Multiple survey Vj images can be replayed in sequence as media images ("animation") that can be displayed on the display device DD. Therefore, the animation may convey a visual sensation that the target anatomical feature is "wobbling" to the user, sufficient to estimate a sufficiently large but tight scanbox from observation of the animation alone. The animation may be replayed more than once, as frequently as necessary, in the playback loop. In this way, the media image Vj displayed as video / animation shows the user how the target anatomical feature TAF moves as it transitions through different motion states, thereby indicating the different spatial portions within the examination area that it is expected to occupy. The display device may be integrated into the operator console OC of the imager IA.

[0091] The facilitator system FS may calculate scan boxes based on such outputted predicted survey Vj without user involvement, particularly without displaying either the calculated prediction or virtual survey Vj in the media / video footage as described above. However, preferably, there is some user involvement through the use of a user interface UI. In such a more user-centric embodiment, one or more predicted survey Vj may be displayed either in conjunction with or instead of the original V0, and the user may use graphic tools, which may be supported by the user interface UI, to enable the user to essentially draw / outline scan boxes using the spatial information from the displayed predicted survey Vj. In particular, the aforementioned media footage / animation may be rendered to a view display device DD to provide the user with a diagram of the portion of space occupied by the target anatomical feature during the transition of motion states.

[0092] The user may stop the animation after one or more repetitions of the video footage at the point where the furthest position in the associated extreme motion state is reached, and then draw a scan box accordingly. The user interface UI may be implemented by touchscreen or pointer tool (mouse, stylus, etc.) interaction. The user may draw a geometric shape, such as a 2D rectangle or a 3D cuboid, such that the boundary or line of the scan box SB is drawn tangent to the spatially most extreme point of the target anatomical feature TAF represented in the predicted extreme survey Vj*, so as to tightly enclose the extreme motion state of the target anatomical feature TAF. The coordinates specified by this drawing action are captured and stored as a scan box specification, which may then be transmitted to the control interface CL automatically or upon request. A diagnostic scan based on the specified scan box may be initiated. Again, the start of this scan may be automatic when the user dispatches the specified scan box, or the scan may be initiated by a go-ahead signal issued by the user. Due to the assumed 3D nature, the video image replacement of the predicted service Vj may be performed along each spatial dimension X, Y, and Z, and in each view, the scanbox specification is made to capture the extreme motion state in all dimensions. The three partial scanbox specifications thus obtained, i.e., one specification for each view along each axis of the coordinate system X, Y, and Z, may be geometrically combined for the complete spatial definition of the scanbox, which is then sent to the control interface CL. Again, a user interface UI with drawing tools may be used for each partial specification along each standard axis X, Y, and Z. For example, a simple 2D display of the predicted service as a still image or animation on the existing display unit of the imager's operating console OC may be sufficient for most purposes.Alternatively, 3D rendering based on algorithms such as marching cubes is used, preferably with rotation of the target anatomical feature in any state, so that the user can define the area around the scanbox in such a 3D view for all three dimensions at once. The dimensional definition in partial specification may not necessarily be based on visualization along the standard axes X, Y, Z. Instead, the view-by-view definition of scanbox SB may be based on a reformatted view, and any new coordinate system with axes X', Y', Z' instead of the standard coordinate system X, Y, Z may be used.

[0093] A freehand drawing tool may be used in the user interface UI, or a guided tool may be used to assist the user in defining a predefined shape of a scanbox. Such a scanbox is rendered based on the user changing its one or two dimensions, allowing the user to center the scanbox accordingly, and the third dimension is dynamically adjusted and scaled to capture extreme motion states (maximum spatial deviation from the initial motion state) in an appropriate predictive survey Vj*. Such a guided drawing tool restricts user input along certain directions to emphasize the geometric configuration (such as angles between edges) of the preset scanbox shape, while automatically completing other directions.

[0094] In this embodiment, Figure 3, which shows a schematic block diagram of the facilitator system FS assumed here, is referenced herein.

[0095] Broadly speaking, input data is received at the input interface IN. The input data includes at least one survey V0. This is processed by the prediction component PC, along with additional information optionally provided by the camera DSC (described later). The prediction component PC may use a motion model M to calculate one or more virtual survey Vj based on the input data, which are made outputtable through the output interface OUT. Based on the output virtual survey Vj, scanbox specifications are acquired and may be passed to the control interface for diagnostic imaging / more detailed but high IQ imaging, with a higher IQ than that of the input survey V0. Scanbox SB specifications may be calculated automatically by the planning module PM, or they may be defined by user interface UI interaction with the user as described above. In this case, the planning module PM may still be used to convert user input, such as the drawing commands described above, into scanbox SB specifications. For example, per-view sub-specifications may be combined by the planning module PM into the required 3D specifications of the scanbox SB.

[0096] A different virtual survey Vj, as an output in the external or internal output interface OUT, preferably has the same size (length, height, width) as the initial input survey V0. The planning module PM may combine information from the virtual survey Vj to calculate the scan box SB as a definition of a geometric shape, such as a cuboid, sphere, or ellipsoid, as described above. In an automated embodiment, the planning module may identify an extreme survey Vj* and tightly fit a geometric shape of a predetermined form to the extreme point of the target anatomical feature estimated in the extreme survey Vj*. Segmentation may be used in conjunction with tangent fitting of a preset shape to geometrically define the scan box. The scan box may be displayed and stored in memory MEM for later use, or passed to the control interface CL described above to control subsequent operations in the diagnostic scan of the imager IA to acquire second high IQ projection data λ in the diagnostic scan of the scan box voxel.

[0097] The newly acquired projection data λ, obtained at a higher quality, can then be reconstructed by the RECON reconstructor into a diagnostic volume V of the target anatomical features with low noise and good contrast. This image can be used for treatment, diagnosis, training, or any other medical task required. The above process flow may be referred to as an automated process, as it does not require user involvement.

[0098] A second embodiment using user involvement via the user interface described above is shown in the lower left of Figure 3, where a graphics display generator GDG displays one or more virtual motion state surveys Vj on a graphics display GD, preferably in video / animation. The display of surveys Vj on the display device is selective, such that one or more of the predicted surveys Vj are displayed either alone or in combination with the original volume V0, and / or two or more virtual surveys Vj, Vj' are displayed as needed. Preferably, the animation is rendered by displaying the virtual surveys VJ in sequence, preferably one in place of the other, in combination with the original volume V0. The display order preferably emphasizes the predicted natural order of the motion states, although this is not necessarily required. Thus, the surveys Vj are either output in the correct order by the prediction component or in any order, and can be rearranged into anatomical order by comparing the target anatomical feature positions by the virtual surveys Vj with the initial positions recorded in the initial input survey V0. The virtual survey Vj that is spatially close to the original is displayed first, followed by virtual survey Vj that are spatially distant. This is to reproduce natural animation that emphasizes anatomical reality. However, with respect to the scanbox specifications, the display order may be sufficient as long as the extreme survey Vj* is identified either in 3D or separately along each dimensional axis as needed.

[0099] Referring more closely to the embodiments of the facilitator system FS, two main embodiments are assumed here and described in more detail below, in terms of input data.

[0100] One such embodiment relies solely on one or more surveys V0 collected from previous measurements (projection data) acquired in exploratory imaging at low doses. Prediction of a virtual survey Vj for other motion states of the target anatomical feature can be performed by a prediction component PC based solely on one or more input survey volumes V0 received as input. However, in another embodiment, which may be referred to as a two-channel embodiment in contrast to the aforementioned one-channel embodiment, there is a second channel (the first channel being one or more input surveys V0) in which video images of surrogate features SF of the patient PAT are acquired by a suitable non-ionizing radiation-based camera DSC.

[0101] The image camera DSC may be mounted in the gantry SG of the imager IA, or within the bore BR, to acquire images of surrounding anatomical structures SF, such as the patient's skin surface. However, such placement within the bore or gantry is not always necessary and can be mounted at any point in the examination room where the imager is installed, so that images are acquired before the patient is introduced into the bore. For example, the camera may be mounted on a stand, wall, or ceiling as needed. The camera DSC may be specially fitted, or an existing such camera may be repurposed in the scanbox design. Such existing cameras may have been used for patient placement or monitoring.

[0102] This time-series video footage (mt) contains dynamic information about how the motion state of the target anatomical feature changes. However, since the target anatomical feature is an internal anatomical structure and the camera DSC is non-ionizing, this information is not directly captured. Instead, the time-series video footage captured by the camera perceives external features, i.e., surrogate features such as the patient's chest, which receive motion states but are different from the motion states of the target anatomical feature of interest. However, the motion states of the surrogate features (SF) may be related to the motion states of the target anatomical feature of interest, and such a relationship may be utilized here in some such two-channel embodiments.

[0103] This survey image V0 and video footage from camera DSC m t Information from (which may include multiple frames m1, m2, m3, etc.) is combined and processed by the prediction component PC to calculate a virtual survey Vj, and based on the virtual survey Vj, the scan box SB can be acquired by the user and / or the planning module PM. As previously stated, the camera can preferably operate based on near-infrared (NIR), non-ionizing radiation such as IR, or optical (visible) light spectrum. A depth-sensing camera may be used to acquire a spatial profile of a surrogate feature SF, such as the chest, and how that motion state changes over time, for example, with respect to a respiratory cycle, which may be associated with the motion state of the lungs or heart. The depth-sensing image can be conceptualized as a 2D set of point measurements, where each pixel represents a depth measurement from the focal spot of the camera DSC sensor along each geometric ray extending from the focal spot to each point on the patient surface, such as a point on the patient's skin or a cloth covering the chest, and different pixels correspond to different such rays and points on the patient. The chest is an example of a surrogate feature SF, where the state of motion serves as at least partial spatial information that is a surrogate for the actual state of motion of the target anatomical feature TAF of interest.

[0104] A depth-sensing camera (DSC), as an embodiment of a non-ionizing camera (DSC), acquires a sequence of depth profiles from a camera reference position on the skin of a patient's PAT, such as the patient's chest. The depth values ​​change over time in response to the patient's movement, particularly the movement of surrogate features, skin pathways, the chest, etc. Therefore, the movement of (internal) target anatomical features that cause an observable motion state of external surrogate features within the patient's PAT, or are otherwise associated with them, cannot be directly observed with this approach, as previously stated. However, it is the motion state of surrogate anatomical features (SR) that is observed instead, without consuming additional dose. For example, a surrogate feature (SF) (skin of the chest) may at least partially surround a target anatomical feature (TAF) (lungs, heart), or may be spatially related in other ways. The movement of the target anatomical feature (TAF) may or may not cause motion of the anatomical feature, as in the case of the lungs and skin of the chest, or the heartbeat and skin of the chest, etc. However, such a causal relationship is not essential, as long as there is some association between the motion (state) of the target anatomical feature and the surrogate feature (SF).

[0105] Therefore, spatial information regarding the movement of a surrogate feature SF, such as the patient's skin near the target anatomical feature, can be associated, and the movement component can be extracted from this information and applied to the original input survey V0 to predict / extrapolate a virtual survey Vj for other movement states of the target anatomical feature TAF. Again, a single survey for an extreme position / movement state may be sufficient here for the defined scan box, but it is preferable here to obtain multiple such surveys, each representing a different intermediate movement state to obtain a more robust and realistic understanding of the movement state of the target anatomical feature. However, extreme movement states should preferably also be included when multiple such surveys Vj are calculated. Again, more robust results can be achieved in this way.

[0106] Now, moving on to a more detailed two-channel embodiment, the predictive component PC may use a more detailed motion model M to calculate a virtual survey Vj for different motion states of the target anatomical feature TAF.

[0107] The motion model M that can be used by the prediction component PC to compute the virtual survey Vj can be of one of two types. The model may be an explicit motion model or an implicit motion model. In an explicit motion model, the motion components are explicitly computed as vectors and can be applied to the original survey image V0 to compute the predicted volume Vj. For example, a single such vector may be used to transform the image representation of the target anatomical feature TAF by the input survey V0. More than one such vector may be computed for different points in three dimensions and applied to different points on the image representation m(TAF) of the target anatomical feature, for example, to perform deformation. Tensor or matrix representations may be used for more complex motion patterns. Rotation matrices, etc., may be used. Segmentation may be used to geometrically define the in-image representation m(TAF) of the target anatomical feature TAF. One or more vectors may be applied to points on the boundary of the in-image representation m(TAF).

[0108] In other embodiments, particularly in one-channel embodiments, machine learning models such as artificial neural networks that do not require specific computation of motion components are used. In such machine learning approaches, the model may be able to extrapolate appropriate patterns learned from a large number of training data instances of known motion patterns and input survey volumes V0 acquired anyway to learn a general relationship between input survey images V0 and motion information. Thus, in this embodiment of machine learning, a sequence of virtual surveys Vj can be computed even based on a single survey input image V0. In some embodiments, more than one survey image V0 may be applied as input. Also, as mentioned above, the machine learning model M may be configured to simply compute one Vj* for a single virtual survey, in particular for an extreme motion state.

[0109] In another embodiment, the machine learning model may be used in a two-channel approach. In this embodiment, the model is used to capture video footage of patient PAT surface SF images acquired by a non-ionizing camera as described above. t This is processed in combination with the survey input V0. This video footage captures the time evolution of a surrogate anatomical feature SF, such as a surrounding feature that may enclose the target anatomical feature TAF. Such surrogate features may include, in particular, the surface of human skin, or any other body part whose movement can be associated with the movement state of the internal target anatomical feature. Model M is trained with training data to learn the relationship between the survey image and the camera footage of the surrogate feature SF. Each frame m received from camera DSC j,t=j Given initial information from the measured input survey V0, a ​​corresponding virtual survey Vj representing the motion state of the target anatomical feature TAF can be calculated. Therefore, conceptually, the survey input V0 can be understood as a regularizer for the database of model M, and the surface image stream from the camera provides motion information. Thus, the input survey V0 and the surface video stream m j,t=jThe combined information helps to "maneuver" Model M to provide the "correct" virtual survey Vj for each or some video frame instant j.

[0110] Moving on to a more detailed explicit motion modeling embodiment, the facilitator FS may include one or more components, such as a calibrator CAL, a registration unit REG, and a motion component extractor MX. Such components may be part of a motion model adapter MP, which is capable of adapting the motion model M to a given patient PAT and the details of the current imaging setup when including the imaging geometric configuration.

[0111] Roughly speaking, the calibrator CAL uses the acquired camera image m t The image is calibrated using the imaging geometric configuration. The registration unit REG is used for image m t The selected motion model M is registered. More specifically, the calibrator calibrates the camera image using the imaging geometric configuration of the imaging device. The camera and CT scanner are typically installed in fixed positions within the scanner room, and their respective distances, angles, and other geometric parameters are known from the system specifications. In addition, the registration procedure by the registration unit REG can be performed during installation and may optionally be checked at periodic intervals. For example, markers placed at known locations on the patient couch PS and elsewhere can be identified in the camera's RGB image and used for this purpose. Other techniques may be employed.

[0112] For explicit motion modeling, motion video is extracted using the motion extractor MX. t This may include the extraction of motion vectors from the video footage by the registration unit REG. tis registered at the timing of. When the motion components (vectors, matrices, tensors, etc.) are appropriately registered temporally and / or spatially, they can be applied to the servo input V0 that functions as part of the model in this case. For example, a vector field may be applied to move the voxels of the servo image V0 in other forms such as expansion or compression, or rotation, translation, etc. This enables extrapolating different motion states. Thus, the virtual servo Vj can be extrapolated for each new video frame mt with one corresponding motion vector v t =v(m t ). At this point, all voxels of the servo V0 can be the target of the application of the extracted motion v t that includes the voxels constituting the inner representation m(TAF) of the target anatomical feature TAF in the image. However, concentrating the application of the motion information v t only on the inner representation m(TAF) of the image of the target anatomical feature TAF can still be done by using a segmenter (not shown). However, for this, an appropriately trained segmenter that can handle low-signal, high-noise data similar to the low-dose servo image V0 may be required. A machine learning-based segmenter trained on ground truth data that may include artificially noise-corrupted training input images can be used. The similarly trained segmenter may also be used by the planning model PM when calculating / fitting the scan box shape around such segmentation s(m(TAF)) of the inner representation m(TAF) of the target anatomical feature TAF.

[0113] In addition, or alternatively, such a trained segmenter can be used in user interface UI-based embodiments to better assist the user in drawing scanboxes sufficiently tightly around a segmented structure s(m(TAF)). With respect to the tightness requirement, a dead space Δ = 0 is preferred here, but this is not necessarily enforced in all embodiments, and a looser dead space requirement Δ' with some appropriately positional error margin may be acceptable, where Δ' > 0.

[0114] As an optional feature, the system FS may include a motion validator VAL. The motion validator VAL can compare the extracted motion to the mode-predicted motion. If a deviation exceeding a preset threshold exists, there may have been abnormal patient motion. This can be flagged, and a control signal may be passed to a transducer TR (such as a display device, speaker, lamp, etc.). The transducer TR may respond by emitting a sensor warning signal to alert the user about potentially abnormal patient motion that may negatively affect IQ.

[0115] The operation of some components of the facilitator FS will be described in more detail with respect to embodiments of explicit motion modeling, in contrast to the implicit modeling (machine learning-based) described later. Explicit motion modeling preferably relates to the motion of surrogate features SF (patient surfaces, such as the skin surface of the chest). The modeled motion of the surrogate features SF may be known to be associated with the motion of the target anatomical feature TAF.

[0116] Figure 4 shows the operation of the registration unit REG and calibrator CAL for explicit motion modeling. While respiratory cycle-based modeling is used as an example, the principles described here are readily applicable to other scenarios.

[0117] Generally, as mentioned above, it is preferable to register motion models in space and time. Spatial registration can be understood as matching which part of the moving portion of the survey corresponds to the motion recorded by the camera, such as a depth-sensing camera. Temporal registration can be understood as the identification of phase information, such as inspiration and expiration or cardiac phase. The latter temporal registration can be achieved by instructing the patient to exhale or inhale before the survey is taken.

[0118] Moving on to spatial registration in more detail, the actual patient position on the couch is captured by the registration unit REG using camera images m t This can be registered. Therefore, registration-based methods can be used for this purpose. For example, specific landmarks LM (eyes, mouth, hands, joints, etc.) of the patient identified in the RGB channels of the video stream can be used to identify the patient's position on or relative to the couch. These landmarks LM can be used to spatially align the patient PAT with an explicit motion model M. Depending on surrogate features SF or target anatomical features TAF, a respiratory prediction model may be used. As an example of a surrogate feature SF, it is preferable to know which 2D region of the D channel (depth channel) of the video stream mt should be used to observe the distance changes caused by respiration. Thus, a target region representing the motion of the surrogate feature SF can be identified in the video.

[0119] A patient-specific respiratory motion model may be selected or generated. The model generator MP may be able to operate to calibrate / parameterize the model M with the patient's actual respiratory cycle over time, or with any surrogate feature SF. For example, the camera's D channel and target region may be used. The minimum and maximum distances of the patient surface SF within the respiratory cycle are identified and used to calibrate the motion model in the time domain. Once the model M is prepared, the motion extractor MX extracts the motion components. The predictive component may apply these motion components to a portion of the 3D survey V0 or to each voxel, or to each or each time point. For example, the translation of a target anatomical feature TAF voxel due to respiration may be calculated using an explicit model to obtain a virtual survey Vj for a given time t = j. This may be repeated over time to generate a sequence Vj that can be displayed as an animation to convey the range of motion and facilitate the definition of a tight scan box that covers the target anatomical feature TAF in most, if not all, motion states.

[0120] Continuing to refer to Figure 4, and by providing details regarding the model creator MP in the context of explicit motion modeling, the patient landmark may be identified using an RGB camera DSC to spatially align the 3D respiratory model with the patient, in particular, to align a target region observed in the camera's D channel for calculating distance changes due to the patient's respiration. Figure 4 illustrates this approach in relation to the target region. The rectangle shown in Figure 4 represents at least a portion of the model M. The rectangle shows the spatial alignment of the region in the D channel ("target region") based on its relationship with the landmark LM identified in the camera's RGB channel. For example, the posterior position is aligned with the patient table PS. The target region is motion information in the image representing the motion state of a surrogate feature SF. Thus, the spatial registration of the motion model (shown by the rectangle in Figure 4) may be based on the landmark identified in the camera's RGB channel. Only the pixels within the rectangle are used to determine the temporal respiratory state. The respiratory state uses the values ​​of the D channel. The use of an RGB-D camera combining optical image and depth information is not required here. Both channels may instead be provided by separate cameras. The optical channel / optical camera identifies landmarks using image recognition techniques, while the depth channel / depth camera allows the surrogate feature SF to re-encode the motion state information. Again, the optical channel is optional and may not be required in some embodiments.

[0121] Instead of using associations with landmarks LM, the input 3D survey V0 itself can be used for registration. The input 3D image V0 provides some tissue and bone structures that can be registered with a 3D atlas of the human body, and the atlas itself can be "elastically" matched using the patient's contours (which can be identified in RGB channels) and the contours of the atlas. In another embodiment, precise spatial alignment of landmarks LM and motion models between camera data and survey data may not be necessary. Instead, a temporal motion model may be extracted from camera data and applied directly to survey data V0 to generate respiration-induced deformations.

[0122] The temporal alignment of the 3D respiratory model may be based on the identification of inspiratory and expiratory motor states. Such inspiratory and expiratory motor states are examples of the extreme motor states described above and may be evident not only in the target anatomical feature TAF but also in the associated motor states of the surrogate feature SF. Preferably, for robustness, several intermediate motor states are also identified.

[0123] Several options exist for selecting an appropriate respiratory model, and accordingly, surrogate (anatomical) features SF are used. Regarding the changes in the motor state of such surrogate features SF due to respiration, the following options may be considered.

[0124] One such model may allow for forward / backward movement, and linear interpolation between maximum and minimum inspiratory / expiratory states (forward) may be identified in the D channel within the target region. The rearward position is fixed.

[0125] Such models can be improved by automatically segmenting 3D survey VO within bone and soft tissue structures. Interpolation can distinguish between rigid and more elastic movable parts within the respiratory cycle.

[0126] The model is preferably spatially aligned with the 3D survey V0 and matched with the respiratory cycle. Therefore, it may be beneficial to record which movement state (inspiration / exhalation) the 3D survey VO was acquired in, as described above. This matching can be easily achieved, for example, when the camera detects respiratory motion even while acquiring the survey.

[0127] In another embodiment, moving on to a more detailed machine learning model-based implicit motion modeling, we have a two-channel embodiment (input data including camera video mt and input service V0), or a one-channel embodiment (input data including camera video m t Various approaches, including survey V0 without, are assumed here. Such ML approaches may use training data. The training system TS schematically shown in Figure 5 is such training data {(x ~ ,y ~ Based on this, a machine learning algorithm may be used to adjust the parameter θ of the model M. The machine learning model may be general-purpose and may not require explicit analytical modeling of the motion components, as described above in the explicit motion modeling-based embodiment.

[0128] For example, in one embodiment, the query function Q is used to search for cases in a medical database such as an HIS (Hospital Information System) or PACS (Picture Archive and Communication System) that contains past patient records of various patients appropriately diversified by age, sex, and demographics, in which survey images containing anatomical structures of interest (TAFs) have been acquired at a given time.

[0129] The imaging setups used in such past examinations may have included the aforementioned video imaging camera DSC, appropriately installed in the examination room, in the gantry, or within the bore, as described above. Such camera DSCs may have acquired images of surrounding anatomical structures SF, such as the patient's skin surface, while acquiring projection data for survey V0. Such cameras may have been used for purposes other than those intended here (e.g., patient monitoring). However, such images may be reused for the current purpose of training a machine learning model M to predict a virtual survey Vj.

[0130] During the acquisition of projection data for the input survey image V0, by acquiring such camera footage of surrogate features SF, such as chest skin, pairs of naturally associated training input data (video footage) and their targets (each survey V0) can be collected. Thus, the reconstructed survey V0 can be associated with specific surface images, such as depth sensing profiles acquired during projection data acquisition, using timestamps. ~ =m t , y ~ =V0) i Such instants i can be supplied as training data to an ML model M for training. Thus, model M can be trained to associate surface images of surrounding structures SF of different patients with different associated instances of survey V0. Thus, a dynamic association between such video footage and the corresponding survey image V0 may be trained. Finding such pairs of surveys associated with surface images from past patient records may be difficult. In such cases, experimental data collection projects at different medical facilities can be set up. Such projects may involve installing cameras on tomography imaging equipment such as CT or MRI and obtaining a sufficiently large accumulation of such training data pairs over time. Alternatively, existing in-bore / in-gantry cameras originally intended for patient monitoring during imaging may be repurposed in this way.

[0131] Therefore, in this two-channel ML approach, the correspondence between the temporal motion state of a surrogate feature SF (such as the surface of the chest) measured by a camera and the visceral motion state of the target anatomical feature TAF measured by survey V0 can be learned from training data. The training data may include the aforementioned pairs of multiple 2D or 3D tomographic images (MRI, CT, ultrasound) and associated camera images acquired during respiration.

[0132] Preferably, regressive machine learning models, such as convolutional or neural network models, can be used to learn this relationship between surface images and associated surveys. Once trained, in the unfolding / inference phase, the machine learning model takes the current survey and the currently acquired surface video frame as input to compute a virtual survey and an initial (projection-measurement-based) survey V0 for a given video frame. Thus, the above is an example of a two-channel application with a machine learning implementation.

[0133] Another approach employs generative machine learning modeling. Such an approach may allow for setting up a single channel where only the input survey V0 is required. A camera DSC is not required here, but can be used if necessary. In this generative ML modeling approach, a database query Q is used to identify historical images containing different types of survey images capturing anatomical structures of interest for various patients. It is unlikely that all these survey images were collected under the same motion state. Therefore, the motion states encoded by these historical surveys V0 can encode different motion states and thus effectively provide a sample of survey images across different demographic layers. This training imagery may be sufficient for the generative ML network to learn a virtual survey image Vj based on an arbitrary input initial survey image V0. In this case, after training, during deployment, the generative model generates different instances of the virtual survey Vj under different motion states of the target anatomical feature TAF using the input survey M(...M(M(V0)) ... ) = V j The model can be repeatedly applied, starting from (index j being the number of times the model M is applied iteratively). Alternatively, the model may be trained to output a predicted sequence of surveys, given the input surveys. For example, a past survey v0 (in lowercase) is given by input x for any given instance j. ~ =v0j may be split into two or more past surveys, target y~ = v0 k , v0 k+1 , ...v0 k+m It may also be used as such. This can be split into different such combinations to form a fairly large training dataset. "m" determines the length of the sequence (animation if necessary) of surveys that you want to predict.

[0134] Generative ML modeling can also be conceptualized as sampling from the underlying probability distribution of the space from which the training data is extracted.

[0135] Generative machine learning models may include GANs (Generative Adversarial Networks), variational autoencoders (VAEs), hidden Markov models (HMMs), and mixed models such as Markov random field (MRF). Some such generative models may be based on convolutional neural networks (CNNs) or ANNs (Artificial Neural Networks), and these models have been observed to produce good results on spatially related data, such as survey images, which are of primary interest here. As an extension or alternative to CNN models, transformer models may be used, along with attention mechanisms in which convolutional filters are adapted to act differently on different parts of the intermediate feature map input.

[0136] Moving on to an embodiment of the GAN-type ML model M, this type of ML model was previously described by Ian Goodfellow et al. in "Generative Adversarial Nets," which was published on June 10, 2014, and is available on the preprint server at arXiv:1406.2661v1.

[0137] GAN-type learning may have two neural network (NN) models: a discriminator network ("D") and a generator network G. The discriminator D may be configured as an NN classifier. The discriminator D may be configured as a deep fully connected network, a deep fully convolutional network, or a hybrid network containing one or more layers of both types.

[0138] During training, the GAN setup operates as follows: The generator G generates a sample survey G(r) from a seed signal (r) applied to it (e.g., random noise). The discriminator D takes the generator output G(r) and the data that can be stored in the training database. 、 The attempt is to distinguish it from the real survey sequence. Ground truth is survey V0, which represents the target anatomical feature TAF in different movement states for different patients, collected from the medical database by query as described above. i It has a past sequence.

[0139] A specially configured cost function is used, and the parameters of the generator G and discriminator D are adjusted based on it. Such a binary cross-entropy cost function was described by Goodfellow et al. (above) in eq(1), page 3. The cost function represents two opposing objectives similar to a zero-sum game: the objective for the discriminator D is to maximize the probability of correct labeling, and the objective for the generator G is to produce an output such that the discriminator D cannot statistically distinguish between the real survey V0 obtained from the training data and the virtual survey generated by the generator.

[0140] The parameters of the two models G and M may be adjusted alternately in separate iterative runs in the minimum-maximum procedure. The iterations converge the parameters of models G and D to a kind of Nash equilibrium, at which point the generator can produce a "fake survey" that looks real. At this point, the discriminator D can no longer distinguish between the fake survey and ground truth, and the discriminator D is no longer needed. Thus, the generator G can be used to generate a virtual survey from a given survey V0.

[0141] Conditional GANs (cGANs) may be used as an extension of GAN-type generative ML models.

[0142] However, generative modeling is not the only option for a single-channel embodiment. In addition, any recurrent ML model setup configured for processing sequential data, such as a recurrent neural network (NN), can be considered. Certain types of recurrent NNs may include LSTMs (Long Short-Term Memory Networks).

[0143] Figure 5 provides a more detailed reference to the training system TS. Broadly speaking, for ML models, three processing phases can be considered: a pre-training / learning phase, an optional subsequent testing phase, and a subsequent deployment / inference phase. In the training phase, the model parameters are adjusted based on the training dataset. A training / learning algorithm may be used to perform the training. In testing, the model's performance is checked based on test data. The deployment / inference phase is used in intended environments such as clinical practice when the deployment data is processed by the model to compute a virtual survey. Test data is different from training data. Also, deployment data is different from both test data and training data. The training system TS performs the training phase. The training phase may be a one-time event or may be repeated using a previously trained version of the model as a starting point. For example, training may be repeated once new training data becomes available.

[0144] The training system TS may be run on one or more computing systems and is configured to train the ML model M. The training data consists of pairs (x) for a supervised setting. ~ ,y ~ ) k It is shown as x ~ This shows the training input image, y ~ indicates the target associated with it, and the index "k" is the index for the training pair, which constitutes the training dataset.

[0145] In the training phase, the architecture of a machine learning model M, such as a CNN, is pre-inputted with an initial set of weights. The weights θ of the model NN are parameterized by M. θ This represents the weights, which may include the filter parameters of the convolution operator CV. The objective of the training system TS is to train the training data (x ~ k , y ~ k ) k The process involves optimizing and thus adapting the parameters θ based on pairs. In other words, learning can be mathematically formulated as an optimization scheme that minimizes the cost function F, although a dual formulation that maximizes the utility function may be used instead. Instead of such an initialized model, a pre-trained model may be used.

[0146] Assuming a cost function paradigm F, this measures the aggregated residuals, i.e., the total error that occurs between the data estimated by the neural network model NN and the target by some or all of the training data pairs k in a batch or across all the training data. argmin θ F = Σ k D[M θ (y ~ k ), y ~ k (1)

[0147] In equation (1), the function M() is equal to the training input x ~ The results of Model M applied to the target y are shown. The results are generally associated with the target y. ~ This is different. This difference, or the respective residual for each training pair i, is measured by the distance measure D[-,-]. Thus, the cost function F can be pixel / voxel based, like a cost function of the L1 or L2 norm, or any other norm Lp. Specifically, the Euclidean cost function in (1) (such as least squares) may be used for the regression task described above when the output layer regresses to one or more virtual surveys. The cost function in (1) is merely exemplary. Other cost functions may be used in the aforementioned GAN or cGAN models, such as cross-entropy based cost functions for the generator G and discriminator D.

[0148] Output training data M(x ~ k ) is the applied input training image data x ~ k The target y associated with ~ k This is an estimate of the value. As mentioned above, generally, for each pair k, this output M(x ~ k ) and associated target y ~ k An error exists between the two. Optimization procedures such as back / forward propagation or other gradient-based methods are considered for the pair (x) under consideration. ~ k , y ~ k ) may be used to adapt the parameters θ of the model M to reduce the residuals for, or preferably the sum of residuals in a batch (subset) of training pairs from the complete training dataset.

[0149] The optimization procedure may proceed iteratively. The model parameter θ is the pair (x) of the current batch. ~ k , y kAfter one or more iterations in the first inner loop, which is updated by the updater UP, the training system TS then sets the next training data pair x ~k+1 , y ~k+1 Alternatively, the next batch is processed as appropriate in a second outer loop. The structure of the updater UP depends on the optimization procedure used. For example, the inner loop performed by the updater UP may be performed by one or more forward and backward passes in a forward / backpropagation algorithm or other gradient-based setup based on the gradient of F. When adapting the parameters, the aggregated, e.g., summed, residuals of all training pairs are taken into consideration. The aggregated residuals can be formed by constructing the objective function F as a sum of squares of residuals, such as equation (1), for some or all of the considered residuals for each batch. Other algebraic combinations are also possible instead of the sum of squares. In general, the outer loop passes batches (sets) of training data items. Each set ("batch") with multiple training data items and the sum of (1) extend to the entire batch rather than iterating through training pairs one by one, but this latter option is not excluded here.

[0150] Optionally, one or more batch normalization operators (not shown) may be used. These batch normalization operators may be integrated into the model M and, for example, coupled to one or more convolutional operators CV within a layer. The BN operator allows for the mitigation of vanishing gradient effects, reducing the gradual decrease in gradient magnitude in repeated forward and reverse passes experienced during the learning phase of gradient-based learning algorithms in the model M. While the batch normalization operator BN can be used for training, it can also be used for deployment.

[0151] The training system shown in Figure 7 can be considered for all learning schemes, particularly supervised schemes. Unsupervised learning schemes can also be assumed here as an alternative embodiment. A GPU may be used to implement the training system TS. A fully trained machine learning module M may be stored in one or more memory MEMs and made available as a trained machine learning model to be used in the system FS.

[0152] In addition to procuring historical survey images, or instead, training data may be artificially generated by a training data generator system TDGS. For example, a generative model such as a GAN may be used and trained on some ground truth data to generate realistic-looking surface images that represent the correct motion state of a surrogate feature SF of interest. Historical surveys can be more readily procured from existing stock. The images thus generated can be paired with each survey to obtain the training dataset {(x,y)} in this case.

[0153] The ML model assumed here may be a parameterized statistical model.

[0154] For example, a series of baseline 3D surveys of volunteers are acquired from MRI at different exercise states throughout the respiratory cycle. By using an image registration method, patient-specific respiratory models can be derived from baseline volunteer images and generalized using other such volunteer studies. In this way, a parameterized statistical model may be obtained.

[0155] Here, Figure 6 is shown, illustrating a flowchart of a computer implementation method for facilitating the collection of planning data, particularly for predicting scanboxes based on survey volume. Planning data can be used to control predictive data acquisition in tomographic imaging.

[0156] In step S610, input data is received. The input data may include a 3D survey volume. The input survey V0 includes a representation of the target anatomical feature or object of interest (OI). This survey is reconstructed from low-dose projection data previously collected by exploratory image acquisition using an imaging device. Therefore, the input survey is based on measurements based on the collected low-dose projection data. When collecting low-dose projection data, the field of view (FOV) is large (not tight) and covers anatomical structures wider than the target anatomical feature (TAF). In particular, the FOV is not tight because the target anatomical feature (TAF) may be affected by motion. In extreme cases, survey V0 may be based on a whole-body scan, or on scanning a wider anatomical cross-section, such as the entire torso, even if the target anatomical feature (TAF) is only one of the structures, such as the lungs, heart, or abdominal structure.

[0157] Typically, a survey covers a large field of view, extending from the neck to the pelvic bones. The survey is a still image acquired before the actual diagnostic procedure. Immediately before the survey is taken, the patient is asked to hold their breath. Therefore, the respiratory state in the 3D survey is known, and this information is used in the following processing.

[0158] In some embodiments, the input data is multi-channel and includes, in addition to the input service V0, collected video camera footage of an external portion of the patient, referred to here as a surrogate feature SF. The camera used to record the footage is an optical camera, or one based on non-ionizing radiation such as IR, NIR, or LIDAR. A depth-sensing camera such as RGB-D may be used to collect a time-series depth profile of the surrogate feature SF. The surrogate feature SF may be the subject of motion, which may be related motion of an internal target anatomical feature TAF. Thus, the video footage records such motion of the surrogate feature SF over time. The surrogate feature SF may include, for example, a patient surface such as skin that at least partially surrounds the anatomical structure TAF of interest. The length of the footage over time is such that, in the case of periodic motion, all expected motion cycles of the target anatomical feature TAF / surrogate feature SF are recorded. However, even for non-periodic motion, the footage should be recorded for a sufficiently long period of time for better robustness to ensure that sufficient motion information is collected. The exact length of the footage depends on the case. The camera may be installed within the scanner room, typically in a position where it can observe the couch and the patient. The video may record part or all of the patient over time.

[0159] In step S620, the input data is processed, preferably based on a motion model M, to compute one or more virtual predictive survey images representing the target anatomical structure in motion states different from the motion state represented by the input survey V0. As previously mentioned, it is technically possible to output a single such volume representing the extreme motion state with the furthest spatial reach compared to the motion state of the target anatomical feature TAF captured by the input survey. However, it is preferable to have an initial motion state by the initial survey V0 and an extreme survey V0. j* Multiple virtual surveys V0 are calculated, including those representing one or more intermediate motion states between the extreme motion state and the resulting extreme motion state.

[0160] As mentioned above, when a two-channel approach is used, step S620 is performed on the video footage of the surrogate feature SF acquired by the camera. t This may include extracting motion components such as vectors, matrices, or tensor forms. One or more extracted motion components may be used to predict different virtual volumes V. j To obtain different motion states, the initial volume V0 can be applied to simulate them, performing, as necessary, translation, rotation, deformation (such as deformation or compression like cardiac motion). Thus, this embodiment is an example of explicit motion modeling. Such an embodiment may include a motion model setup phase, including calibration of the image and the geometric configuration / survey V0 of the imager IA. Model creation may also include registration, segmentation, etc.

[0161] In some embodiments, a single-channel approach is employed, where only the survey input image V0 is fed to the model to predict one or more virtual survey Vj representing motion states different from those of the initial volume. This single-channel approach may employ implicit modeling, such as a machine learning model. This model may be generative, predicting a sequence of virtual survey Vj. The modeling may be based on a neural network, particularly a CNN. Recurrent models such as LSTMs may also be used. Model building in this case involves using ML machine learning algorithms to tune the parameters of the ML model based on the training data.

[0162] Input service V0 and surface image m t The two-channel approach using this method may be used in conjunction with machine learning models as described above.

[0163] The calculation of the virtual survey may be performed sequentially all at once, or it may be synchronized with the timing of video acquisition. Therefore, video frame m t In some or each instance t, the corresponding virtual survey is computed, mt,t=j → This is Vj. Therefore, there may be some delay until all virtual survey Vj are calculated over a sufficient period of time. This per-video-frame calculation of virtual survey Vj may be used for explicit modeling or implicit modeling (ML-based modeling). Information from more than one video frame may be used to calculate any given virtual survey Vj.

[0164] In an optional step (not illustrated), the motion of surrogate features extracted from the video is compared to a motion model. In the validation step, deviations between the model prediction and the actual recorded video may be observed. If such deviations exceed a threshold, this fact may be flagged as an anomaly, which may indicate unintended patient movement. A warning may be issued to the user. Visual, tactile, or acoustic transducers TR may be used to issue the warning. Thus, video acquisition may continue during image acquisition for a diagnostic scan to act as a motion guardian against unintended patient movement that could cause motion artifacts.

[0165] In step S630, the predicted virtual survey Vj is output and made available for further processing.

[0166] For example, in step S640, the scanbox is calculated based on one or more projected survey Vj. At any given geometric or irregular shape, such as a convex structure, the tight interface is formed to include all the different parts occupied by the target anatomical structure of interest represented in the predicted survey VJ.

[0167] The scan box may be calculated automatically. The scan box may undergo further processing, such as storage, or it may be used to control diagnostic acquisition in step S650. The scanbox calculated here is patient-specific and takes relative patient movement into account. The calculated scanbox can be displayed on the console in addition to the scanbox already manually defined by the technician.

[0168] In another embodiment, the predicted virtual survey Vj is displayed sequentially as an animation in step S660. Instead of, or in addition to, the playback of such animation, one or more selected virtual survey Vj are displayed before or after, as necessary, in combination with the initially survey V0 that is optionally entered. Based on the display of the virtual survey Vj as one or more still images or as a time-series animation (video), the user can use the graphical user interface or any other user interface in step S670 to partition the scan box based on the displayed survey V0, Vj.

[0169] In step S660, an animation on a virtual service Vj, which is a video showing the transition between different motion states of the target anatomical feature TAF, may be played. The user can easily identify the extreme motion state with the largest positional deviation compared to the motion state recorded in the initial input image V0, which can be used to define a tight scanbox boundary in one or more spatial views. The scanbox boundary may be selected tangentially from multiple views so as to be in contact with the surface of the target anatomical feature represented in the extreme motion state service Vj*. This defines a tight scanbox so as to include all motion states but not any irrelevant voxels that the target anatomical feature TAF never visits. This allows for dose saving and good image quality.

[0170] The above scanbox definitions may be performed successively in different views in the X, Y, and Z directions, and these may be combined to ensure that motion in different, for example, all, spatial dimensions is considered.

[0171] Therefore, steps S660 and S670 represent a user-input-based method for defining the scan box, while in step S640 of a fully automated embodiment, the scan box is calculated without such a display based on a virtual survey Vj. Segmentation of various in-image representations m(TAF) of the target anatomical feature TAF may be used in manual and / or automated embodiments.

[0172] In step S640, even if the scan box is calculated automatically, a virtual image Vj, either a still image or an animation, may be displayed as additional information. The visualization of the calculated scan box may be displayed alone, or preferably in combination with the initial survey V0 or the virtual survey Vj. For example, the scan box may be visualized as an overlay or 3D rendered on the initial survey V0 or on any one or more of the virtual survey Vj, particularly on the extreme virtual survey Vj*. Various methods of visualization and display are preferably used in conjunction with the user input-based embodiments by steps S660, S670. However, this can also be used in automated embodiments where the scan box is calculated based on the predicted virtual survey Vj.

[0173] Here, we refer to Figure 7, which shows a flowchart of a machine learning model-based embodiment (implicit motion modeling). In such a model, motion information is implicitly learned from patterns in the training data, rather than through explicit analytical modeling.

[0174] In step S710, training data is acquired.

[0175] In step S720, the machine learning model is adapted based on the training data, and in step S730, the thus trained model is made available for deployment or testing, such as in clinical practice. The input video V0 and optionally the video encountered during deployment are different from the training data.

[0176] The acquisition of training data S710 may be performed synthetically using a generative machine learning model, or by querying existing medical data records. The latter may include queries against patients, the patient's records being large enough to include the target anatomical features of interest, for each survey x ~ =V0 is included. Preferably, such past recordings may further include the aforementioned images of appropriate surrogate features that can be associated with the movement of the target anatomical feature.

[0177] In yet another embodiment, the acquisition of training data may include placing a surface camera within the bore of the tomography imaging device to acquire, for example, a corresponding surface image of the patient's chest or any other body part, in parallel with projection data onto the survey image V0.

[0178] The collected video may be associated with different instances of the overall patient survey image V0.

[0179] In yet another embodiment, survey images from different patients are collected under the assumption that some or each represent different motor states. Any one of the collected surveys may be used as a training input, and one or more of the remaining surveys may represent target sequences that can be used to train a generative, preferably conditional, model, such as a cGAN, based on the input survey V0.

[0180] Generative ML models include GANS, preferably conditional GANs, such as cGANs and VAEs.

[0181] A supervised training scheme using labeled data, or an unsupervised or self-supervised training scheme, may be used.

[0182] Any of the above system FS and methods can be equally applied to other tomography imaging modalities, such as MRI, SPECT / PET, or other nuclear imaging modalities, where tomography raw data (e.g., projection data) is acquired or requires to be acquired. Similar to the case of CT modalities, a first low-IQ scan may be used in non-CT modalities as well, to obtain imaging plan data (such as scan boxes), which is then used to control a second high-IQ scan. The low-IQ scan may have a larger volume / area / FOV than the volume / area / FOV for the later high-IQ scan. For example, in MRI, survey V0 may have lower resolution, thicker slices, a lower signal-to-noise ratio, but a larger field of view, and may correspond to a faster scan than the diagnostic scan, in order to obtain the overall picture needed to plan the diagnostic scan.

[0183] The components of the facilitator system FS may be implemented as one or more software modules that run on one or more general-purpose processing units (PUs), such as workstations associated with the imager facilitator system FS, or on server computers associated with a group of imagers.

[0184] Alternatively, some or all components of the facilitator system FS may be configured in hardware as a appropriately programmed microcontroller or microprocessor, such as an FPGA (Field Programmable Gate Array), or as a hardwired IC chip, application-specific integrated circuit (ASIC), integrated into the imaging system IA. In further embodiments, the facilitator system FS may be implemented in both software and hardware.

[0185] Different components of the facilitator system (FS) may be implemented on a single data processing unit (PU). Alternatively, some or more components may be implemented on PUs of different processing units, perhaps remotely located in a distributed architecture and connected via an appropriate communication network, such as a cloud environment or client-server configuration.

[0186] One or more of the features described herein can be configured or implemented as, or using, and / or in combination thereof, a circuit encoded in a computer-readable medium. The circuit may include discrete and / or integrated circuits, systems on a chip (SOC), and combinations thereof, machines, computer systems, processors and memory, and computer programs.

[0187] In another exemplary embodiment of the present invention, a computer program or computer program element is provided, which is configured to perform a method step of a method according to one of the above embodiments on a suitable system.

[0188] Accordingly, the computer program elements may be stored in a computer unit which may be part of an embodiment of the present invention. This computing unit may be configured to perform or induce the execution of the steps of the method described above. Furthermore, the computing unit may be configured to operate each component of the apparatus. The computing unit may be configured to operate automatically and / or to perform user orders. The computer program may be loaded into the working memory of a data processor. Accordingly, the data processor may be equipped to perform the method of the present invention.

[0189] This exemplary embodiment of the present invention encompasses both computer programs that use the present invention from the outset and computer programs that modify an existing program to use the present invention through means of updating.

[0190] Furthermore, the computer program element may provide all the steps necessary to satisfy the procedure of the exemplary embodiment of the method described above.

[0191] According to a further exemplary embodiment of the present invention, a computer-readable medium such as a CD-ROM is presented, and the computer-readable medium has computer program elements stored therein, which are described in the preceding section.

[0192] Computer programs may be stored and / or distributed on suitable media (in particular, non-temporary media, but not necessarily) such as optical storage media or solid-state media supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.

[0193] However, computer programs may be presented on a network such as the World Wide Web and can be downloaded from such a network into the working memory of a data processor. According to a further exemplary embodiment of the present invention, a medium for making a computer program element available for download is provided, and this computer program element is configured to perform a method according to one of the aforementioned embodiments of the present invention.

[0194] It should be noted that embodiments of the present invention are described with reference to different subject matter. In particular, some embodiments are described with reference to method-type claims, and other embodiments are described with reference to apparatus-type claims. However, those skilled in the art will understand from the above and below descriptions that, unless otherwise notified, any combination of features belonging to one type of subject matter, as well as any combination of features relating to different subject matter, are also disclosed in this application. However, all features can be combined to provide a greater synergistic effect than the simple sum of the features.

[0195] Although the present invention has been illustrated and described in detail in the drawings and the foregoing description, such illustrations and descriptions should be considered illustrative or descriptive and not limiting. The present invention is not limited to the disclosed embodiments. Other variations of the disclosed embodiments can be understood and achieved by those skilled in the art in carrying out the claimed invention, based on an examination of the drawings, disclosure and dependent claims.

[0196] In the claims, the word “have” does not preclude other elements or steps, and the indefinite article “a” or “an” does not preclude plurality. A single processor or other unit may perform the functions of several items mentioned in the claims. The mere fact that certain means are described in different dependent claims does not imply that combinations of these means cannot be used advantageously. Any reference symbols / characters used in the claims should not be construed as limiting scope.

Claims

1. In a system that facilitates the planning of raw data acquisition using a tomography imaging device, An input interface capable of receiving an input 3D survey image of a movable object occupying a first portion of 3D space, wherein the survey image is previously acquired by a tomography imaging device based on raw data acquired at a lower quality than intended for planned raw data acquisition, and the survey image represents a first motion state of the object's motion. A prediction component capable of predicting one or more predicted 3D survey images that can represent the object in one or more other predicted motion states and the object in one or more predicted motion states occupying one or more other parts of space, based on at least the input survey image, To facilitate the acquisition of the planned raw data, an output interface capable of providing at least one of the predicted survey images is provided. A system that has

2. The system according to claim 1, further comprising a planner module capable of calculating planning data based on one or more predicted 3D survey images.

3. The planning data includes designating a target portion of the 3D space, which includes the one or more other portions of the space, and the target portion of the space is a subset of the input survey images. The system according to claim 1 or 2.

4. The system according to any one of claims 1 to 3, comprising a graphics display generator capable of generating a graphics display for display on a display device, wherein the generated graphics display can visualize one or more different motion states of the object based on one or more predicted 3D survey images.

5. i) The graphics display includes a video that can transition between different motion states, thereby representing an animation of the object occupying different parts of space, and / or ii) The system according to claim 4, wherein the system has a user interface that enables a user to specify the target portion in space based on the graphics display.

6. The system according to any one of claims 1 to 5, wherein the prediction component can compute one or more predicted 3D images based on i) additional input data including video footage from which surrogate features of the object can be acquired by a non-ionizing radiation-based camera, and / or ii) a motion model for the motion of the object.

7. The system according to claim 6, wherein the motion model is a machine learning model trained on training data.

8. The system according to claim 6, wherein the motion model represents at least one motion component of the motion that can be extracted from the video footage, and the prediction component extrapolates the predicted 3D survey image by applying the extracted motion component to the survey image.

9. The system according to any one of claims 1 to 8, comprising a transducer capable of issuing a warning signal if the video image of the non-ionizing radiation-based camera shows a deviation between the captured motion state and the model, wherein the deviation violates a predetermined tolerance condition.

10. A system according to any one of claims 1 to 9, and an imaging configuration comprising one or more of the tomography imaging device, the camera, and the display device.

11. In a computer-aided implementation method that facilitates the planning of raw data acquisition using a tomography imaging device, A step of receiving an input 3D survey image of a movable object occupying a first portion of a 3D space, wherein the survey image was previously acquired by a tomography imaging device based on raw data acquired at a lower quality than intended for the planned raw data acquisition, and the survey image represents a first motion state of the object's motion. The steps include predicting, based at least the input survey image, one or more predicted 3D survey images representing the object in one or more other predicted motion states and one or more objects in one or more predicted motion states occupying one or more portions of space, To facilitate the acquisition of the planned raw data, the step of providing at least one of the predicted survey images, A computer implementation method having

12. A computer implementation method for training a machine learning model used to plan the acquisition of raw data by a tomography imaging device based on training data, wherein the model, when trained, is given a survey image of a moving object in a first motion state and can predict one or more predicted survey images in one or more different motion states.

13. A computational implementation method for acquiring training data to be used in a method for training a machine learning model used to plan the acquisition of raw data by a tomography imaging device based on acquired training data, wherein the model, once trained, can be given a survey image of a moving object in a first motion state and predict one or more predicted survey images of the object in one or more different motion states.

14. A computer program element configured to cause at least one data processing system to perform the method according to any one of claims 11, 12, or 13, when executed by the data processing system.

15. A computer-readable medium storing the program elements described in claim 14, or at least one model described in any one of claims 6 to 8.

Images