Methods and systems for supporting imaging operations of an imaging device

Abstract

System (SYS) for supporting imaging operations of an imaging apparatus (IA) capable of presenting different imaging geometrical configurations and related method. The system comprises an input interface (IN) for receiving a current image of a current region of interest (ROI_1) acquired by the imaging apparatus (IA) at a current imaging geometrical configuration (p). A pre-trained machine learning component (MLC) is configured to compute output data representative of an imaging geometrical configuration change (Δp) for a next imaging geometrical configuration (p') related to a next region of interest (ROI_2). An output interface (OUT) is for outputting technical parameters of the imaging geometrical configuration change.

Description

Technical Field

[0001] The present invention relates to a system for supporting image-based navigation, a method for supporting image-based navigation, a computer program unit, and a computer-readable medium. Background Technology

[0002] In certain medical interventions, such as percutaneous coronary intervention (PCI), clinicians sometimes need to introduce one or more medical devices or tools, such as guidewires, into the patient.

[0003] Interventions can be performed under X-ray imaging guidance, where one or more images are sometimes acquired as a series by an imaging device (“imager”) to form a video feed, which is displayed in real time on a display device for the clinician (referred to herein as the “user”).

[0004] The displayed images allow users to consider lesions, organs, introduced devices / tools, or regions of general interest (“ROIs”). For optimal diagnostic or treatment outcomes, ROIs need to be visualized in the appropriate orientation, which requires adjusting the imaging geometry of the imager.

[0005] During these sometimes challenging and demanding interventions, clinicians may need to switch from one Region of Interest (ROI) to another, which may require readjusting the imaging geometry of the imager. For example, a clinician might begin treating a stenosis in one branch of the coronary artery and then switch to another branch that must be treated at the bifurcation. Or the treatment plan may require switching from one organ to a completely different organ in a completely different part of the body. For instance, a patient might want to image the femoral artery in the groin for better catheter insertion and then later want to image the heart while treating the stenosis.

[0006] However, current interventional X-ray imaging systems do not have "awareness" of the patient's position on the imaging table or the body part currently being imaged.

[0007] The user is responsible for moving the examination table (“examination”) on which the patient lies during the intervention and / or moving the gantry of the imager (such as a C-arm) so that the relevant region of interest (ROI) is within the (potentially relatively small) field of view (FoV) of the imager’s detector. Additionally or alternatively, the appropriate C-arm angle may need to be adjusted to obtain clinically optimal images. Currently, these imaging angles must be manually selected by the operator, thus increasing procedure time and X-ray dose.

[0008] Optical cameras are used only to a limited extent to support image geometry adjustments because they may not be able to detect patient posture on the examination table, where patients are typically covered by sterile coverings and are only partially visible. This lack of "geometric awareness" of the patient's anatomy present on the examination table could, for example, lead to unwanted collisions between a moving C-arm and the patient.

[0009] Adjusting the image geometry (especially when it needs to be done multiple times) is cumbersome and time-consuming. For example, table or C-arm movements take time and increase the overall procedure duration. Typically, X-ray imaging continues during table movements to monitor the current FoV and allow the operator to determine if the desired imaging ROI has been reached, thus increasing the total dose exposure to the user and patient during the procedure.

[0010] Manual or motor-assisted movement of the examination table needs to allow for precise positioning, and therefore allows for slow movement, which makes it more time-consuming to perform larger examination table movements efficiently. Even with variable movement speeds (e.g., accelerating over time up to a maximum speed), the examination table movement process is not optimized because the system still does not know which body part the user wants to image next, or where that (next) region of interest is relative to the imaging system's coordinate system.

[0011] In addition, different imaging settings (“EPX”) may be required for different body parts to achieve optimal dosage and image quality. These settings must be manually changed by the operator once the table / C-arm has been moved. Summary of the Invention

[0012] Therefore, an imaging support system may be needed to address at least one or more of the aforementioned drawbacks in current imaging systems.

[0013] The objectives of this invention are achieved through the subject matter of the independent claims, wherein other embodiments are incorporated in the dependent claims. It should be noted that the aspects of this invention described below are equally suitable for methods, computer program units, and computer-readable media for supporting image-based navigation.

[0014] According to a first aspect, a system is provided for supporting imaging operations of an imaging device capable of presenting different imaging geometric configurations, comprising: An input interface for receiving the current image of the current region of interest acquired by the imaging device at the current imaging geometry; A pre-trained machine learning component configured to compute output data representing an imaging geometry change for a next imaging geometry configuration relating to a next region of interest; and The output interface is used to output the technical specifications of the changes in the imaging geometry configuration.

[0015] Specifically, the input image is the currently acquired live image. The change in imaging geometry can be explicitly provided, which is preferred, or it can be implicitly provided by providing technical parameters for the new imaging geometry. In the latter case, a transformation operation may be required to obtain the technical parameters for the change in imaging geometry. The estimation of the change in imaging geometry implicitly provides a pose estimate of the ROI for the patient on the examination table. Therefore, this estimate can be used to optionally construct an anatomical model including the pose of the ROI for a given patient.

[0016] In a general sense, technical parameters may include one or more data points (such as coordinates, angles, or translations, or combinations thereof) that define a trajectory for the imaging geometry configuration components of the imager. Changing the imaging geometry configuration along this trajectory allows the current imaging geometry configuration to be adjusted to a second imaging geometry configuration. More than one such trajectory may exist, and in practice, the trajectory is sometimes modified to avoid collisions with the patient.

[0017] The specification of image geometrical changes can be implicitly represented by the coordinates of one or more subsequent ROIs (Regions of Interest) relative to the coordinate system. ROI pose coordinates can be converted into image geometrical transformation parameters or into a new imaging geometrical configuration. These parameters specify how the current imaging geometrical configuration needs to be changed so that the next image of the next ROI can be acquired with the calculated pose.

[0018] In other words, given a current ROI image, the proposed system predicts the relative pose of that ROI with respect to one or more other ROIs (also referred to as "signs" in this document), from which the relative pose can be used to derive the relative changes in imaging geometry. In other words, each imaged ROI has a known relationship with an implicit reference coordinate system in patient space, and this implicit reference coordinate system is encoded in a model of a machine learning component. This encoding can be provided by network weights, such as in a neural network model. In one embodiment, the ROI coordinates can then be correlated with changes in imaging geometry, such as in C-arm space. Alternatively, the ML model can directly encode changes in imaging geometry without referencing the ROI pose coordinates.

[0019] In one embodiment, the system includes a control module that instructs one or more actuators based on the technical parameters to cause the imaging device to present the next imaging geometry configuration.

[0020] In one embodiment, the system includes a user interface configured to allow a user to specify the next region of interest.

[0021] In one embodiment, the system includes a graphics generator for generating visual indications of the technical parameters on a display device.

[0022] The visual indications can be adapted to guide the user to cause the imaging device to present the next imaging geometry configuration.

[0023] In an embodiment, the imaging geometry configuration includes any one or more of the following: i) at least one orientation of the X-ray source of the imaging apparatus, ii) at least one orientation of the inspection table of the imaging apparatus, iii) the state of the collimator of the imaging apparatus, and iv) the distance between the detector of the imaging apparatus and the X-ray source.

[0024] In an embodiment, the machine learning component is configured to calculate the image acquisition settings of the imaging device associated with the next region of interest and / or the imaging operation.

[0025] In one embodiment, the machine learning component includes a neural network.

[0026] In this embodiment, the neural network is a convolutional neural network. Alternatively or additionally, the neural network includes a fully connected network.

[0027] In another aspect, a system is provided for training a machine learning model to support imaging operations of imaging devices capable of presenting different imaging geometric configurations, the system being configured to: Receive training input data, which includes training images representing regions of interest and associated imaging geometric configuration changes related to one or more other regions of interest; The training data is applied to the model to obtain training output data; and The parameters of the model are adjusted based on the training output data.

[0028] In this embodiment, the model has a neural network architecture.

[0029] In another aspect, a method is provided to support imaging operations of an imaging device capable of presenting different imaging geometric configurations, comprising the following steps: Receive the current image of the current region of interest acquired by the imaging device at the current imaging geometry configuration; Output data is computed based on pre-trained machine learning components, the output data representing changes in imaging geometry configuration for a next imaging geometry configuration related to the next region of interest; and Output the technical parameters of the changed imaging geometry configuration.

[0030] In another aspect, a method is provided for training a machine learning component to support imaging operations of an imaging device capable of presenting different imaging geometric configurations, the method comprising the following steps: Receive training input data, which includes training images representing regions of interest and associated imaging geometric configuration changes related to one or more other regions of interest; The training data is applied to the model to obtain training output data; and The parameters of the model are adjusted based on the training output data.

[0031] In another aspect, an image processing apparatus is provided, comprising: The system according to any of the above embodiments; and The imaging device.

[0032] In another aspect, a computer program unit is provided, which, when run by at least one processing unit, is adapted to cause the processing unit to perform the method according to any of the above embodiments.

[0033] In another aspect, a computer-readable medium is provided that stores the program unit or a pre-trained machine learning component used in the system as mentioned above in any of the embodiments.

[0034] This paper presents a system and method for learning a model of patient posture on an examination table based on one or more training X-ray images of patients from a patient population, and thus a model for changing the required imaging geometry. Training can be refined by incorporating background images or non-image data from the image learning process. Background data may include previous patient information derived from hospital IT systems such as Radiology Information Systems (RIS), Electronic Medical Records (EMR), and / or Picture Archiving and Communication Systems (PACS). The model thus trained is then used to assist or automate examination table / C-arm movements and / or angulation, and automatically optimizes the EPX (or other) settings for the current body part to be imaged.

[0035] "Imaging geometry": In fluorescence fluoroscopy or other X-ray or non-X-ray imaging modes, imaging geometry can be determined by one or more (in any combination) or all of the following: rotation angle ("rotation"), rotation angle, translation, etc. Typically, imaging geometry involves the geometry of the imager. Imaging geometry involves any setup or configuration that alters the position and / or orientation of the optical axis, the imaginary axis connecting the X-ray source and the X-ray detector, and / or particularly, the setup / configuration of the imaging mode / apparatus that otherwise spatially affects the imaging FoV relative to the object being imaged (such as collimator setup). Imaging geometry may also include SID (source-to-detector distance), thus (inherently) determining the magnification.

[0036] "User": as used in this article, refers to the person operating the imaging device.

[0037] "Patient / Object": can be human, animal or plant, microorganism (e.g., in microscopic imaging), but can also involve inanimate "objects", such as in baggage screening, non-destructive material testing, etc.

[0038] Typically, a "machine learning component" is a computerized arrangement that implements or facilitates the realization of a "machine learning" ("ML") algorithm. A machine learning model can be based on an ML "model." An ML component is configured to perform a task. In an ML algorithm, task performance measurably improves as more and more training data is fed to the arrangement. The model is tuned based on the training data. Performance can be measured through objective testing when test data is fed to the trained model. Performance can be defined by requiring a certain error rate to be achieved for a given test data. See T. M. Mitchell of" Machine Learning Page 2, Section 1.1 McGraw-Hill , 1997. The main task of interest in this paper is navigation support for imaging, particularly the estimation of changes in imaging geometry.

[0039] “Posture” refers to one or both of position and orientation. Attached Figure Description

[0040] Exemplary embodiments of the invention will now be described with reference to the following figures, which are not to scale, wherein: Figure 1 It is a block diagram including the imaging arrangement of the X-ray imaging device; Figure 2 It is a virtual model representing the spatial relationships between different symbols; Figure 3 This is a schematic block diagram of a computer-implemented navigation system for imaging support; Figure 4AA schematic diagram of the machine learning component is shown; Figure 4B It is a computer-implemented training system used to train machine learning components; Figure 5A This is a flowchart of a method supporting imaging operations; and Figure 5B This is a flowchart of the method for training machine learning components. Detailed Implementation

[0041] refer to Figure 1 The diagram illustrates an image-based navigation support device (AR) for use, preferably in the context of medical intervention.

[0042] The AR device includes user-operable X-ray images of the patient's internal structures at the region of interest (ROI_1). I i Imaging apparatus IA, particularly X-ray imaging apparatus. Region of interest ROI_1 can be the human heart, lungs, or another organ or group of organs.

[0043] image I i (Sometimes referred to as frame sequences in this document) can be displayed to the user in real time as a moving picture or video feed on a display device DD, or can be displayed separately as still images as needed.

[0044] The imaging device AR also includes an image processing system SYS to assist the user in adjusting the current imaging geometry configuration (used to image the current ROI_1) to a new imaging geometry configuration to image the new ROI_2.

[0045] As mentioned, in the embodiments, the imaging device AI and the imaging processing system SYS are primarily envisioned herein as supporting medical interventions such as percutaneous coronary intervention (PCI). Other medical interventions not necessarily relating to human or animal hearts are also envisioned, as are non-medical applications, such as image-based support for inspections and work performed in inaccessible caves or piping systems, or for the inspection of technical equipment such as engines and other complex machinery that cannot be directly inspected by the naked eye but requires visual inspection via video feed from imaging devices to make occluded areas of interest accessible.

[0046] Now let's first refer to the imaging device IA in more detail, which can be as follows: Figure 1 An imaging device arranged as a C-arm or U-arm type, as shown in the exemplary embodiments. Figure 1In this embodiment, the C-arm system IA is mounted on the ceiling CL, but this may not be the case in all embodiments. Optionally, the imaging device IA is mounted on the floor, or on a bracket, etc. In another alternative, the imaging device can be mobile, such as wheeled or track-mounted.

[0047] An X-ray imaging apparatus includes an X-ray detector D and an X-ray source XS. In general, in embodiments, but not necessarily in all embodiments, the imaging apparatus includes a gantry G that carries the X-ray detector D and the X-ray source XS (such as an X-ray tube). The X-ray detector and the X-ray source XS are arranged on the gantry G in a relative spatial relationship to form an examination area between the X-ray source and the X-ray detector. The patient PAT is located in this examination area, such that the region of interest is approximately located at the isocenter IS of the imaging apparatus. The patient can lie on the examination table TB during imaging. The examination table TB can be adjusted in height H, can be translated along the X-axis or Y-axis or both X-axis and Y-axis, and in embodiments can also be tilted about one or two tilt axes.

[0048] During the imaging process, an X-ray source XS is excited by applying cathode current and voltage across the anode and cathode to generate an X-ray beam XB emanating forward from the focal point of the anode. The beam exits the X-ray source, passes through the examination area, and thus through the patient tissue at and around the region of interest, and then irradiates the X-ray-sensitive surface of an X-ray detector D. The X-ray-sensitive surface of the detector D may include pixel elements that convert the irradiated X-rays into intensity values. These intensity values ​​can vary with location due to differential attenuation of the X-ray beam caused by locally different material densities in the tissue or tissue type.

[0049] The intensity values ​​recorded at detector XS can be mapped to image values ​​based on a color or grayscale palette to form a projected image (“frame”). Acquisition circuitry operates to capture a sequence of different projected images at different times in this manner at a suitable frame rate during the imaging process. An exemplary frame rate contemplated herein is 20–30 fps. For example, in fluorescence fluoroscopy, as the primary mode contemplated herein, intensity values ​​can be mapped across a range from black to gray to white, where darker image values ​​correspond to lower intensity values. Other mapping schemes, such as inverse mapping, can be used, where lower intensity values ​​are mapped to brighter image values, as commonly used in radiography. Other mapping schemes can be used alternatively.

[0050] The spatial width of the primary X-ray beam defines the field of view (FoV) of the imager IA. Objects residing in or extending into the field of view and thus into the X-ray beam will modify the intensity of the locally detected X-rays at the detector. The field of view can be changed upon user request or automatically by adjusting the imaging geometry of the imager IA, such as by moving the X-ray source, moving the patient, or by amplifying or limiting the beam width using a collimator (not shown), or a combination of all or any subset of the aforementioned components. The system SYS is configured to support changes in imaging geometry, as will be described more fully below.

[0051] The X-ray detector can be arranged as a digital flat panel detector communicatively coupled to a display device D. The flat panel detector D can be of the direct conversion or indirect conversion type. In an alternative embodiment, the imaging detector can be arranged as an image intensifier coupled to the display device via a camera.

[0052] Although attenuation is the primary mechanism for imparting contrast to projected images as envisioned herein, other imaging techniques that utilize additional or alternative contrast mechanisms, such as phase contrast and / or dark-field imaging, are not excluded. In the latter two cases, the imaging apparatus may include additional components, such as interferometers.

[0053] The imaging apparatus includes a control console (CC) through which the user can determine when to start and stop the imaging process, specifically when to excite the X-ray source (XS). A pedal can be coupled to the control console as a user interface to control the power supply to or off of the X-ray source, or to operate a grid switch to stop or resume exposure to the X-ray beam.

[0054] The main propagation direction of the main X-ray beam (leaving behind scattered radiation) is defined by the optical axis OX, which is an imaginary line traveling from the focal point (not shown) of the X-ray source to the center of the X-ray sensitive surface of the X-ray detector D. The optical axis defines the spatial projection direction.

[0055] To better support the user during navigation, the position or spatial orientation of the optical axis can be changed upon user request, and thus the position or spatial orientation of the projection direction. In one embodiment, this can be achieved by arranging the frame to be rotatable about one or preferably two corresponding axes perpendicular to each other. Having two such axes of rotation allows for two degrees of freedom for changing the optical axis. For example, in one geometric configuration, one of the axes of rotation extends to... Figure 1 In the attached plane, and allows the optical axis to rotate about angle β. Another axis of rotation is parallel to... Figure 1 The attached plane, and allows for orientation changes independently of β around another angle α, such as Figure 1 This is illustrated schematically. By convention, the axis with respect to α is defined as "rotation", while the axis with respect to β is defined as "revolution".

[0056] Optionally, the height of the rack itself can also be changed, such as by... Figure 1 The double arrow H indicates this. Additionally, the optical axis OX can be translated by moving the gantry accordingly along the line. The position and orientation of the optical axis can be referred to herein as at least partially defining the imaging geometry. In other words, the imaging apparatus contemplated herein allows the user to change the imaging geometry. Changes to the imaging geometry can be requested by the user operating a joystick or other suitable user interface IG-UI, which is coupled to a console CC. Changes to the imaging geometry can be requested entirely automatically by the proposed computerized imaging support system SYS without any user intervention. Alternatively or as an additional option, the computerized imaging support system SYS can allow the user to change the imaging geometry themselves via the user interface IG-UI, but the imaging support system SYS usefully intervenes, for example, by guiding the user toward the imaging geometry. Guidance can be achieved purely through visual cues, or alternatively or proactively, for example, mechanically or otherwise through tactile stimulation, as will be explained in more detail later.

[0057] Requesting a change in imaging geometry may involve applying a control signal to a suitable actuator AC located at the imaging apparatus (such as at the gantry, inspection table, collimator COL, if present, etc.). The actuator AC operates in response to the control signal to change the imaging geometry. The actuator AC is powered by a power source or manually powered by the user via a handwheel, lever, or other means. The actuator AC can be fully automatic, hybrid, or semi-automatic. In semi-automatic operation, the user operates a user interface such as a joystick or other control device, but the change in imaging geometry may be assisted by a servo motor or similar means.

[0058] The actuators may or may not be encoded. If they are encoded, they may include linear and / or angular encoders, such as potentiometers. Due to the encoders, changes in image geometry implemented by one or more actuators are traceable, i.e., can be mapped to numerical coordinates that change with the imaging geometry configuration.

[0059] Other options for altering the imaging geometry may include changing the detector-X-ray source distance and / or changing the distance between the region of interest and the X-ray detector, and thus between the region of interest and the X-ray source. The latter can be achieved by changing the height H of the examination table TB on which the patient lies. Changing the height h and / or the source-detector distance can be equivalent to rescaling the image with a specific magnification factor. Other options for altering the imaging geometry may include the operation of a collimator (not shown) to restrict or expand the cross-section of the X-ray beam in shape or size to alter the field of view (“FoV”). Yet another option for altering the imaging geometry may include in a plane parallel to the surface of the examination table... X, Y The patient table TB is moved horizontally in one direction, parallel to the direction of movement. Figure 1 The image plane is in one of the attached planes, and extends in another direction into the image plane. The inspection table may also be tilted about one or both axes. Components of the imaging apparatus that participate in the change of imaging geometry may generally be referred to herein as imaging geometry components, and particularly include any, more than, or all of the following: source XS, gantry, detector, collimator (if any), inspection table, etc.

[0060] The angle values ​​α and β mentioned above can be collectively referred to as angle imaging geometric configuration parameters in this paper. p ω The imaging geometry configuration of the inspection stage TB (i.e., translation along x, y or x and y) can be referred to in this paper as the translation imaging geometry configuration parameter. p T It should be understood that the angle imaging geometric configuration parameters p ω It may also include other such parameters (such as table tilt) related to the settings of other imaging geometry configuration components, and / or translation imaging geometry configuration parameters. p ω This may also include settings such as collimator openings or translation settings of the C-arm (such as height H adjustment (e.g., translation along the Z-axis) or translations of other components that determine the imaging geometry). In short, the imaging geometry can therefore be formalized as a typically high-dimensional vector. =( p ω , p T Each imaging geometry configuration is a pose, and it relates to the pose of the corresponding ROI that can be imaged as the corresponding imaging geometry configuration. Changes to the imaging geometry configuration may include only translating the imaging geometry configuration parameters. p T Changes, or only the angle imaging geometry configuration parameters p ω Changes, or p ω andp T The variations, particularly in some simpler embodiments, involve changes to the imaging geometry configuration that only affect the inspection table TB or C-arm gantry in the x, y or x and y Movement in both directions. Angular changes (if any) are manually performed by the user. In other embodiments, as needed, [further details are needed]. p ω and p T The changes in both may only be p ω Changes in imaging geometry configuration. It should be understood that the above definitions of imaging geometry configuration components / parameters are in the context of C-arm X-ray imaging; this disclosure is not limited to this mode, and the principles explained herein can be applied to any other imaging apparatus that allows for changes in imaging geometry configuration. Of course, according to p ω , p T The parameterization will be different.

[0061] Typically, changes in imaging geometry alter the spatial relationship between the X-ray source and / or detector relative to the region of interest. Alternatively or concurrently, the field of view can be altered by collimator movement or by moving the patient (e.g., by TB translation of the examination table as described).

[0062] In a typical imaging protocol, a user can excite an X-ray source and acquire an image or image sequence I (ROI_1) of a first region of interest (ROI_1) at a first imaging geometry. The acquired frame(s) are displayed on a display device DD as a video feed or still image. The user can then request to stop imaging, i.e., the X-ray source is powered off or the X-ray beam is otherwise disabled (e.g., by collimator or gate switch operation).

[0063] Then, the imaging protocol may need to acquire images of the second region of interest (ROI_2) at different imaging geometry configurations. In fact, some imaging protocols, such as PCI, may require the user to access multiple (more than two) different ROIs in a sequence, and acquire corresponding images or image sequences at each ROI with a different corresponding imaging geometry configuration. Especially when such complex imaging protocols are required, the computerized navigation support system SYS proposed in this paper helps the user access ROI_1 through ROI_2. K（When K>1), different imaging geometry configurations are automatically or at least semi-automatically implemented to quickly and reliably find those ROIs in sequence. Alternatively or additionally, the system SYS assists the user in operating the imaging geometry configuration user interface of the imaging device by providing visual cues of how the imaging geometry configuration should be changed or by mechanical engagement with the user interface, thereby facilitating correct user interface operation.

[0064] The basic operation of the proposed navigation assistance system SYS can be found by referring to... Figure 2 To illustrate, a schematic diagram of the patient's "virtual" model M is shown in a two-dimensional view.

[0065] The crosshairs indicate the location of different regions of interest for a specific intervention, such as the heart (H), the carotid artery (A), the kidney (L), and the femoral entry point (R). Furthermore, in... Figure 2 In the scenario shown, the current femoral entry point R is designated as the first or current region of interest (ROI_1), which is imaged to acquire the first or current image I (ROI_1) at the correct imaging geometry configuration. Next, the protocol may require imaging the heart H, which will then necessitate a change in imaging geometry configuration. This change in imaging geometry configuration can be conceptualized and is practically defined as transforming the current geometry at ROI_1 to that at ROI_2 = ... H The geometric transformation of the updated imaging geometry configuration required for proper imaging of the heart.

[0066] This transformation from R to H, or more formally... T R,H This indicates a change in imaging geometry, which can be achieved through angular or translational motion, or a combination of both. Specifically, this transformation is achieved or caused by one or more actuators AC of the imager IA, which drive corresponding imaging geometry components, such as translation of the inspection table and / or rotation / turning of the C-arm. In an embodiment, the proposed system SYS is configured to control one or more actuators AC accordingly.

[0067] It should be understood that some imaging geometry configuration changes involve both the moving patient examination table TB and the gantry G (e.g., the C-arm of an interventional X-ray imaging system). In embodiments, both movements are actuated and encoded. The relative orientation of the gantry C and the examination table TB at least partially or completely determines the current imaging geometry configuration of ROI_1. Therefore, changes in imaging geometry configuration specifically include relative changes, where one or more imaging geometry configuration components are actuated to alter their orientation.

[0068] For example, the relative movement between the C-arm G and the inspection table TB can imply that only the inspection table TB or only the C-arm G, or both components, move to change their relative orientation, such as position.

[0069] More formally, let T T To check the 3D attitude transformation of the inspection platform in a fixed world reference coordinate system, and T C Let's consider the 3D attitude transformation of the C-arm within the same fixed reference coordinate system. Then the relative motion (or attitude transformation) is: T R = T C -1 · T T (1) Expression (1) describes the relative motion between the inspection table and the C-arm, where " -1 " indicates the inverse of the transformation, and "·" indicates the multiplication / cascade of transformations. In the following, each imaging geometry configuration transformation may be described as a single transformation matrix with entries associated with a common fixed world coordinate system or a combination of two (such as in (1)) or more transformation matrices, each matrix Tj representing the attitude change of the associated imaging geometry configuration component j.

[0070] It should be understood that the model M envisioned in this paper does not explicitly encode each ROI location, or more generally, its pose. This is because there are no absolute values ​​to describe this, as the exact pose of the corresponding region of interest may vary from patient to patient, and this, in turn, directly implies how the imaging geometry configuration needs to be modified. Instead, this paper proposes to implement model M as a machine learning model, as part of a machine learning component MLC, which is trained on training data comprising patient images from historical cases or synthetically generated images. The parameters of the machine learning model M are then adjusted based on the training data. These adjusted parameters then collectively and implicitly encode the mutual spatial relationships between regions of interest as a whole patient population rather than individual patients. The following will... Figure 4A , 4B and Figure 5B This paper provides a more comprehensive explanation of the suitable machine learning model proposed in this paper and the method for training the machine learning model based on training data.

[0071] Now let's turn to the operation of the image processing system SYS, and now refer to... Figure 3 The block diagram explains this in more detail. In summary, the navigation support system (SYS) is configured to perform imaging geometry configuration changes or to assist the user in achieving the correct imaging geometry configuration.

[0072] The imaging support navigation system (SYS) is computerized. It can be implemented on a single or multiple data processing systems (PUs). The SYS is integrated into the imaging device (IA), or otherwise communicates with the supply frame via wired or wireless means. I i The imaging device is grounded and coupled to the IA communication.

[0073] The system SYS can be arranged as one or more software modules appropriately linked and running on one or more processing units. Alternatively, the imaging processing system SYS can be arranged in hardware as a appropriately configured microcontroller or microprocessor. In an alternative, the system SYS is implemented in both hardware and software. The system can be implemented partially or completely as a microcontroller, optionally fully integrated into the imaging device IA. Some or more of the components of the image processing system SYS described below (particularly the machine learning component MLC) can reside in one or more memory SMs appropriately communicatively coupled. The data processing system PU can be a appropriately programmed general-purpose computing device. The processing unit may include a graphics processing system (GPU) for rapid computation.

[0074] Continue and refer to more detailed information. Figure 3 This shows a schematic block diagram of the components of the proposed navigation support system SYS. System SYS includes a machine learning component, MLC. It is now assumed that the machine learning component has been fully trained on suitable training data. Therefore, system SYS will now be explained during the "deployment phase" (i.e., after the "learning / training phase"). However, it should be noted that the training of the machine learning component may not necessarily be a one-time operation, but rather allow the system to learn from new data, especially when it discovers it is performing poorly for some reason. These repeated learning phases (using different training data than before) allow, for example, the system SYS to be tailored to the specific characteristics of a patient population, such as in a medical setting.

[0075] In the deployment phase described here, the patient PAT is positioned on the examination table TB, and a first / or next image I (ROI_1) of the ROI_1 of interest is obtained by operating the imager IA at the first imaging geometry configuration. This first / current image I (ROI_1) is received at the input port IN of the navigation support system SYS. The image can be preprocessed by an optional preprocessor module PP. Preprocessing may include cropping, resampling, scaling, histogram normalization and other normalization or standardization steps, and any one or more of various types of data augmentation (rotation, translation, scaling, and addition of anatomical structures or interventional devices obtained from simulation). Optionally, a vector with additional patient biometric data (such as height, weight, BMI, age, sex) is used. a k It is also preprocessed (e.g., normalized, embedded) and combined with the current live imaging data. I kTogether, they are provided to the pre-trained MLC. Although the use of live image data is preferred, alternatively or otherwise, supplementary image data, such as one or more preoperative reconnaissance images, whole-body CT images, MRI images, ultrasound images, or images of other modalities, may be used.

[0076] The system SYS operates under the control and coordination of the application controller ACO. It may preprocess live input images (a single image may suffice) along with optional (potentially preprocessed) background data. a k Together, they are provided as live input to the machine learning component, MLC. The MLC, including trained parameters (such as NN weights), is stored in one or more storage media, SM. Under the control of the application controller, the MLC is used to compute an estimated, preferably normalized, pose vector based on the live input. p k The attitude vector p k Define the required imaging geometry configuration change for the new ROI / marker (ROI_2) to be imaged. When estimating the imaging geometry configuration change, the response time of the machine learning component MLC has been observed in the fractional second (e.g., millisecond) range, especially when using one or more GPUs.

[0077] The Imaging System Controller (ISC) is used to change the normalized imaging geometry configuration vector. p k This is converted into command or control signals for one or more actuators AC of the associated imaging geometry configuration (“IG”) component. For example, the estimated normalized position vector. p k It can be converted into a relative checkerboard motion transformation T from the current ROI_1=H to one or more flags ROI_2=I. R , iControl signals can specify, for example, relative table motion to properly image the target ROI_2. Actuator control signals determine a trajectory in phase space. This trajectory is typically the shortest possible path that allows the imager to rapidly adapt to the new imaging geometry. Sometimes, the shortest possible trajectory may lead to patient collisions. To avoid this, the imaging system preferably includes a collision avoidance system (“CAS”) with one or more distance / proximity sensors (such as capacitive sensors) suitably arranged on a movable IG component. The imaging control system can interface with the CAS to modify the estimated trajectory to avoid collision events. While the proposed system SYS can be used with such sensor-based CAS, distance-independent CAS can also be envisioned due to the anatomical awareness of the system SYS. This collision avoidance “bias” or property can be implemented during the training phase by providing imaging geometry changes as targets designed to avoid collisions in the first place.

[0078] The system may include one or more user interfaces (UIs), either graphical or non-graphical, configured to allow users to define the target ROI / target anatomical structure to be imaged next (e.g., i="H (heart)"). Specifically, graphical rendering of model M and / or marker I can allow users to efficiently select the next / desired imaging location via voice control or touchscreen actions. Once the user specifies target i, the MLC returns an estimated IG change for target region ROI=i. In other embodiments, given the current ROI, the MLC immediately calculates all IG changes for all other ROIs. However, the relevant IG information is only displayed or forwarded to the imaging system controller (ISC) when the user specifies the next ROI_2=i via an input operation of the user interface.

[0079] In an embodiment, the MLC can provide the imaging geometry configuration change vector without explicitly specifying it and natively. p kFurthermore, the system SYS can provide its output based on the attitude specification of the target imaging geometry configuration itself for the next region of interest (ROI)_2. In this case, the system SYS may include an attitude processing module PPM, which calculates the IG change Δp as Δp = p - p' given the current IG p and the estimated new IG p'. For example, the required imaging geometry configuration change Δp is calculated by the attitude processing module PPM based on the current imaging geometry configuration p (e.g., the current table position and C-arm position / rotation and / or rotation) and the estimated new imaging geometry configuration p' (which is related to the patient and landmark posture). Additionally, the attitude processing module PPM can be configured to convert ROI attitude coordinates into imaging geometry configuration parameters when the MLC supplies its output not directly based on the imaging geometry configuration parameters but based on the attitude coordinates of one or more ROIs. In this case, the attitude processing module PPM is used to convert a (normalized) attitude (particularly position) vector p, such as that supplied by the MLC, into an absolute or relative imaging geometry configuration change, such as a required table motion transformation T to one or more ROIs i. R , i .

[0080] By way of example, given the current image of the current ROI_1, the user can use the user interface (UI) to define the target ROI_2 / target anatomical structure (e.g., "heart") to be imaged next. Based on the current table and / or C-arm position (or posture) and the estimated patient and landmark ROI posture, the relative table motion and / or C-arm rotation and / or swivel required to image the target ROI are calculated in the posture processing module (PPM).

[0081] Estimated imaging geometry configuration change vector p k And / or the target pose can be visualized in graphical and / or text form on a display device DD by a graphics generator (GG). For example, the displayed information can guide the user to the next inspection station location.

[0082] In some embodiments, the Imaging System Controller (ISC) automatically changes the imaging geometry configuration by controlling the associated actuators (AC) without further user intervention. For example, the surgical table (TB) movement and / or C-arm rotation / turning can be implemented without user interaction. In other embodiments, the ISC can control the speed of the table / C-arm movement based on the estimated distance to the target, allowing for faster movements over greater distances and slower movements for fine-tuning closer to the target ROI_2.

[0083] Alternatively, this paper also envisions the passive or semi-automatic application of the imaging system controller ISC to assist IG changes, and also envisions the effects of other actions that are beneficial to the effective imaging of the next target ROI_2.

[0084] In a passive embodiment, the graphical display generated by the graphics generator (GG) on a display device can be configured to indicate the current and target coordinates of the examination table and / or C-arm. The display can also graphically show the current positions of the examination table, C-arm, and estimated patient posture, as well as the target ROI and the corresponding target table / C-arm position. These indications can be displayed in text / numerical form or via directional or rotational graphical widgets such as arrows. The displayed navigation indications can be updated when the user and / or system controls the movement of IG components such as the examination table and / or C-arm.

[0085] In another passive embodiment, additional visual or audio feedback related to the remaining distance to the target ROI can be provided. The intensity or other quality of the visual or audio feedback can be modulated proportionally to the remaining distance and / or proportionally to the deviation from the estimated target IG.

[0086] Semi-automatic embodiments of the Imaging System Controller (ISC) may include circuitry engagement with the IG-change user interface (IG-UI) to modulate mechanical or other effects that guide or induce the user to achieve an estimated IG change.

[0087] Typically, semi-automatic embodiments of the system SYS provide machine-guided / assisted but user-executed system localization. This can be implemented in embodiments with haptic feedback or orientation compliance modes, i.e., by limiting the possible trajectories of the system, allowing the user to easily change the imaging geometry configuration from ROI_1 to ROI_2, for example, using a single control. This keeps the user “in the circle” (which may be required by local safety regulations) and still achieves similar workflow benefits to the described automatic localization.

[0088] For example, in one embodiment, the controller ISC may restrict motion to the shortest / most direct trajectory from the current imaging geometry to the target imaging geometry, or at least cause greater resistance to motion when the user operates the control user interface IG_UI, so as to cause divergent erroneous movements away from the predetermined trajectory.

[0089] In other embodiments, the imaging system controller (ISC) induces haptic feedback by controlling the user interface (IG-UI) to guide the user to the shortest trajectory. The haptic feedback is modulated based on the distance in the phase space between the current IG and the target IG, for example, achieving lower intensity / frequency vibrations when the user conforms to a predetermined trajectory and higher haptic intensity / frequency when the trajectory diverges. Additionally, the resistance to the mechanical operability of movable portions of the user interface IG-UI (e.g., a joystick) can be modulated via electromechanical engagement. For example, the joystick IG-UI can be operated with less force when it follows a trajectory conforming to an estimated change in imaging geometry.

[0090] Now refer to Figure 4 for a more detailed explanation of the machine learning component MLC and its training aspects during the training phase.

[0091] First, refer back to the above combination. Figure 2 The concept or virtual model MD discussed implicitly encodes one or more landmarks in the patient model M. Preferably, the landmarks correspond to the center / centroid of an organ (such as the heart, brain, etc.) that is frequently imaged during interventional X-ray imaging, or the center of another landmark (such as a vertebra), where their associated positions are identified relative to the origin of a two-dimensional (2D) or three-dimensional (3D) coordinate system.

[0092] Conceptually, when using machine learning methods as envisioned in this paper to estimate changes in imaging geometry, the task can be formalized as follows. There often exists an unknown and potentially highly complex latent mapping that, for any patient ω, maps the (current) imaging geometry configuration for the current ROI_1 to another imaging geometry configuration for another region of interest ROI_2.

[0093] This latent mapping can be formalized as: L :( P, ω) ( P', ω) (2) This latent mapping operates not only on the phase space (the space of all coordinates), but on a more abstract, enhanced space P x ​​ω, where the patient characteristic ω adds additional dimensions.

[0094] Therefore, the set of adjusted parameters (such as weights in a neural network model) of the machine learning model under consideration can be considered as an approximation of the aforementioned latent mapping L.

[0095] In other words, the parameters θ of the ML model M are partially or approximately "encoded" based on the estimates from the training data. Figure 2The virtual model MD is shown. The coordinate formulas do not need to be displayed. The corresponding spatial relationships between regions of interest across the considered patient population ω are encoded into the learning parameters θ of the machine learning model.

[0096] The machine learning model envisioned in this paper is of the regression type, in which an input image, optionally enhanced using non-image data such as imaging setup and / or patient characteristics, is regressed to an associated imaging geometry configuration change vector. p k However, other regression algorithms derived from "classical" statistics could potentially be used, but they are expected to provide a lower degree of generalization than machine learning methods based on the training data.

[0097] As previously described, according to one embodiment, the machine learning model M used has a neural network (NN architecture), specifically as now referenced. Figure 4A The Convolutional Neural Network (CNN) architecture shown.

[0098] Broadly speaking, the neural network structure of a machine learning component comprises multiple nodes that are at least partially interconnected and arranged in different layers. Layers are arranged in one or more sequences. Each node is an entry capable of hypothesizing values ​​and / or producing outputs based on inputs received from one or more nodes in earlier layers.

[0099] Each node is associated with a function, which can be a simple scalar value (node ​​weight) or a more complex linear or nonlinear function. A "connection" between nodes in two different layers means that a node in a later layer can receive input from a node in an earlier layer. If no connection is defined between two nodes, the output of one node cannot be received as input by the other. Nodes produce their outputs by applying their functions to the inputs. This can be implemented as multiplying the received input by the node's scalar value (weight). The interconnected hierarchical nodes, their weights, layer sizes, etc., form an NN (neural network) model, as an example of the ML model envisioned in this paper. The model can be stored in memory as a matrix or tensor structure. Once trained, this structure forms a trained machine learning component that can be maintained on one or more memory locations (SMs).

[0100] Figure 4A It is used to input images from (live) sources. I k Or the sequence inference imaging geometry configuration change vector of such images p k An exemplary embodiment of a suitable CNN configuration. The model is preferably a deep neural network including one or more hidden layers. L iThese are hidden layers because they are positioned between the input and output layers. The network M consists of several convolutional filter layers. L i The composition consists of one or more convolutional filter layers employing one or more convolutional filter kernels. Each or some convolutional filter layer is followed by an activation layer, and in some embodiments, a pooling layer (not shown). Optionally, one or both batch normalization and random deactivation layers are also present. Further optionally, one or more fully connected layers are also present. The above series of layers terminates at the generation (preferably normalized) vector. p k The estimated regression layer. In some embodiments, a classification output is alternatively envisioned.

[0101] Activation layers define a (typically non-linear) function that modifies the output values ​​from one layer and feeds them as input to the next layer. Pooling layers combine the outputs of multiple elements from one layer to serve as input to a single element in the next layer. Each convolutional, activation, and pooling layer works together to process data non-linearly, changing its dimensions and passing it to the next layer. During the training phase, the parameters of each convolutional layer are "learned" (optimized). The number of parameters in each convolutional layer depends on the size of the convolutional kernel, the number of kernels, and the stride ("stride") when moving across the image being processed at a given layer. The number of parameters in a fully connected layer is determined by the number of elements in the previous and current layers.

[0102] Optionally, additional input parameters a k These non-image parameters—typically categorical, sparsely represented data—can be represented using low-dimensional embeddings (e.g., one-hot encoding). For example, such embeddings can be processed by one or more fully connected layers and then added pointwise or cascaded to arbitrary intermediate feature maps (layers) by tiling the outputs from one or more fully connected layers along the spatial dimension of the feature map.

[0103] Once trained, during deployment, real data (such as in the current image geometry) I k [or( I k , a k The current real-time input image at a given ROI_1, captured at [location 1], is applied to the input layer INL. Image data I k Through the hidden layer L1- L N The application of the filter is represented as propagating from the input layer INL. Layer L1- L NThe quantity is on the order of two, three, or ten (e.g., 20-50 or others). Image data I k It is transformed during propagation and then appears as the feature vector layer OUTL. Background data a k It can propagate through different threads (not shown) of a model that includes one or more fully connected layers. These two threads (processing image data) I k Thread L1- L N and processing (potentially non-image) background data. a k The output of the thread can be merged, as will be explained in more detail in the following reference to the embedding.

[0104] The Feature Vector Layer (OUTL) can be considered as the input image. I k The feature vector OUTL is then followed by one or more task-specific layers (not shown), such as fully connected layers or convolutional layers with task-dependent activation functions (e.g., any one or more linear, sigmoid, softmax, tangent hyperbola). The task is regression or classification. One or more task-specific layers are applied to the feature vector layer OUTL to estimate the imaging geometry configuration change technique parameters in an embodiment regressing to the imaging geometry configuration change technique parameters. The imaging geometry configuration change technique parameters are formalized herein as the desired final output. As mentioned earlier, instead of a regression task, classifications are generated based on directional qualifiers ("left", "right", etc.).

[0105] One, more than one, or all hidden layers L i These are convolutional layers, and their operations can be defined based on filters. The convolutional layers used in model M are distinguished from fully connected layers. In a convolutional layer, each node is connected only to a subset of the (pooled) nodes in the previous layer. Nodes in subsequent layers are computed by forming filtering operations based on sub-selection of nodes from the previous layer. For different nodes in subsequent layers, different subsets from the previous layer are preferably processed using the same filter. Filters are digital arrays that can be arranged in a matrix. f ij The set. In some embodiments, the filtering operation includes forming a product. f ij n jk The sum, n jkThe nodes are from the previous layer. Therefore, this product-sum operation is similar to a traditional convolution or filtering operation, where the filter mask slides across the matrix data. Typically, layers are implemented as matrices. The filtering operation that propagates data through layers is implemented as a matrix operation and is preferably run on a GPU or other processor capable of parallel processing, such as those with multi-core designs. In fully connected layers, each node from the previous layer is connected to a node in the current layer using learnable activation weights.

[0106] As mentioned above, this paper also envisions methods for processing non-image data. a k Fully connected layers (such as the patient's biological characteristics). Fully connected layers forming separate threads (not shown) are envisioned, wherein one, two, or more such fully connected layers are arranged sequentially, and non-image data... a k The data is passed through a separate thread of the processing layer. Optionally, the data is also passed through a separate and different convolutional layer from the convolutional layer used for image processing. The output of this non-image data processing thread can then be fed into... Figure 4A In the graph processing thread, specifically, the output of the last of those fully connected layers used for non-imaging data can be cascaded to one of the hidden layers in the CNN branch of the model, thus merging the non-image data with the image data. In this way, the output (i.e., the change in imaging geometry) takes into account not only the spatial data in the image data but also the background data that provides more detail about the patient, thereby enabling the model to generalize better to different types of patients.

[0107] As mentioned above, embeddings such as one-hot encoding can be used to represent sparse categorical data as vectors. Non-image data processing threads can include a combination of fully connected layers and convolutional layers. Furthermore, as... Figure 2 The image data processing thread shown may include a sequence of one or more fully connected layers, especially when non-image data... a k Combined processing and used as augmented data ( I k , a kWhen passed through these layers, in addition to convolutional and fully connected layers, one or two threads can also include any combination of pooling layers, activation layers, random deactivation layers, etc. Additionally, deconvolutional layers can exist, which can be considered as an approximation of the inverse operation of a convolutional layer. The number of nodes produced at each layer can vary in dimension, so the size (number of rows and columns) and / or depth of the output layer can grow or shrink. The dimension of the output of a convolutional layer is typically the same as or smaller than the dimension of the input, depending on its "stride," which represents how many nodes the filter slides across for each operation. Therefore, downsampling can occur. Upsampling with deconvolutional layers is also envisioned. Some architectures involve one or more runs of two layers, with an upsampling layer followed by downsampling, or vice versa. Typically, Figure 4A The architecture is net downsampling because the network acts as a filter to take the input image. I k Downmapped to vector The mapping.

[0108] Now, referring to the training phase or process, this can be... Figure 4B The training system TS shown is used for implementation. Training is based on suitable training data. Training data can be obtained by acquiring real images or by obtaining simulated X-ray images, both of which are denoted in this paper as... I k Real X-ray images can be retrieved as historical image data from patient records such as PACS or from other image repositories.

[0109] More specifically, training involves acquiring real or simulated X-ray images. I k ( k ∈ 1… K ), which contains the element { p k,1 , …p kN The "baseline truth" coordinate vector of} p k The pairings use normalized coordinates to describe the position of the image relative to one or more of the model markers. The number of elements N depends on the dimension of the model coordinate space (2D or 3D), the number of markers in the model, and the parameterization of the marker positions. Typically, N The range is between 2 and 30, k is large (e.g., k >> 30), and the image I k Selected from a large and variable group of objects. Alternatively, benchmark truth data. p kInstead of being expressed in terms of ROI attitude coordinates, these are instead provided directly as image geometry (variation) coordinates based on any other attitude technical parameters, such as table translation, C-arm gantry rotation / turning, or related imaging geometry configuration components. Imaging geometry configuration p k The data can be expressed as changes in imaging geometry configuration relative to a given ROI, or as absolute imaging geometry configuration parameters for each ROI given a reference imaging geometry configuration (such as an inspection table at “zero” angle / rotation, position / tilt “zero”, etc.).

[0110] Using { during training I k , p k}, where the image I k The coordinate vectors are fed into a convolutional neural network (CNN) and the network parameters are optimized to infer the coordinate vectors that will be the output. p k Optionally, the image can be combined with an additional patient-specific parameter vector as input. a k (Such as height, weight, age, gender, BMI, anatomical abnormalities, implanted devices, etc.) pairing. A suitable CNN configuration for this task is Figure 4A The CNN configuration shown is not the only one presented in this paper; however, other architectures are envisioned, as are models other than neural networks, such as support vector machines (SVM), decision trees, random forests, etc.

[0111] Continue and refer to more detailed information. Figure 4B Computerized training system TS can be used to train data ( X, Y ) i Train the machine learning model M. "X" represents a suitable image. I k "Y" indicates an associated change in imaging geometry that may be enhanced by the image setup data EPX. p k In this paper, it is also referred to as the target of X.

[0112] Training data pairs can be obtained by human experts during manual annotation exercises. Alternatively, and preferably, annotations are retrieved automatically. Imaging geometry configuration data and / or EPX data associated with a given training image can be assembled from metadata such as DICOM header data, or retrieved from patient records stored in a database. This can be accomplished using appropriately programmed scripting tools to find the data for each image. X = I kThe associated imaging geometry configuration change Y= p k .

[0113] Therefore, training data is organized in pairs i, especially for supervised learning schemes as primarily envisioned in this paper. However, it should be noted that this paper does not exclude unsupervised learning schemes.

[0114] Choose a machine learning model M (such as...) Figure 4A The CNN network shown may include methods for processing image data. I k CNN threads and those used for processing non-image data a k (A fully connected layer hybrid network), and the weights are pre-filled with some initial, possibly random or uniform, data. For example, the weights of all nodes could be filled entirely with "1" or other numbers.

[0115] The weights θ of model M represent the parameterization of M. θ Furthermore, the purpose of training the system TS is based on the training data (X). i , Y i ) The optimization is performed on the cost function and thus the parameter θ is adjusted. In other words, learning can be mathematically formalized as an optimization scheme where the cost function is minimized, but alternatively, a double formula for maximizing the utility function can be used.

[0116] Now assume the cost function f This example measures the aggregated residual, that is, the error between the data estimated by model M and the target i based on some or all of the training data: argmin θ f = (3) More specifically, the image data X of the training pair is propagated through the initialized network M. The first pair of "i" X is received at the input IN, passed through the model, and then received at the output OUT as the output training data. M θ (X Use appropriate metrics || || (such as p-norm, difference of squares, etc.) to measure actual output M θ (X The difference between the output Y and the expected output Y.

[0117] Given input training image data X, the output training data M(X) is an estimate of the proposed changes to the imaging geometry. Typically, there is an error between this output M(X) and the relevant target Y currently considered. The model M can then be tuned using optimization schemes such as back / forward propagation or other gradient-based methods. θ The parameter θ is used to reduce the number of parameters considered ( X, Y) i The residual.

[0118] After one or more iterations in the first inner loop, which updates the model's parameters θ based on the optimization scheme used, the training system TS enters the second outer loop, where the next training data pair X is processed accordingly. i+1 Y i+1 This inner loop can be implemented using one or more forward and backward passes in a forward / backward propagation algorithm. However, this time, when adjusting the residuals, not only the individual residuals of the current pair are adjusted, but also, if necessary, the aggregated, summed residuals of all training pairs considered up to that point in time are adjusted to improve the objective function. Aggregated residuals can be formed by configuring the objective function f as the sum of squares (or other algebraic combinations) of some or all the residuals considered for each pair (such as in equation (3)).

[0119] For all learning schemes (especially supervised schemes), consider the following: Figure 4B The generalization training system shown is illustrated. It is also conceivable to envision unsupervised learning schemes that perform appropriate clustering (with or without enhancement) in phase space.

[0120] Now for reference Figure 5A The flowchart in the document explains in more detail the method used to support image-based navigation. This method can be understood as the basis for the operation of the aforementioned image processing system SYS. However, it will also be understood that... Figure 5A The methods and steps explained herein may not necessarily be the same as those described above. Figure 1 -4 relates to the architecture of the image processing system SYS. More specifically, the methods described below can be understood as their own teaching. Assuming the machine learning model has been fully trained, the explanation... Figure 5A The flowchart is as explained above in Figure 4.

[0121] In step S510, the patient is positioned on the X-ray imaging examination table TB.

[0122] At step S520, the first / next X-ray live image of the first ROI_1 is obtained and received at the first imaging geometry configuration IG1 (e.g., the first inspection stage position).

[0123] In step S530, a pre-trained machine learning component (such as the one mentioned above) is used. Figure 4A The CNN at the location processes the first / next X-ray image to infer the desired imaging geometry configuration change (such as relative table motion) from the current imaging geometry configuration at ROI_1=R to a new imaging geometry configuration IG2 for one or more other ROI_2=i or (one or more) model flags. This imaging geometry configuration change can be caused by geometric transformation T. R,i (It is represented by a function of two ROIs (ROI_1=R and ROI_2=I).

[0124] At step S540, the desired imaging geometry configuration change is made available in coordinate vectors. p k Further processing of the form of the coordinate vector p k For example, specifying angle or translation values, or both, for one or more imaging geometry configuration components.

[0125] At step S550, a second (target) ROI is determined by the user or automatically by the MLC based on background data (such as technical parameters of the imaging protocol to be used) and corresponding to any of the labels inferred by the CNN. This selection step S550 for the second ROI (ROI_2) can be performed alternately before step S530.

[0126] At optional step S560, the relative table motion required to center the next image on the target ROI is calculated based on the MLC output and the current table position. This step may be necessary when the MLC output does not explicitly represent IG but rather the new imaging geometry configuration IG2.

[0127] At step S570, changes in imaging geometry (such as table movement toward the target) are implemented automatically or semi-automatically or otherwise facilitated. Examples include automated motion or assistance provided, such as visual or audio guidance, or mechanical feedback (e.g., with tactile assistance or other forms of feedback).

[0128] At optional step S580, any imaging settings (e.g., EPX) are automatically updated for target ROI_2, as specified by, for example, a predefined lookup table or as learned by MLC.

[0129] It is possible that, at step S530, imaging geometry configuration changes have not yet been calculated for all relevant imaging geometry configuration components. For example, at step S530, only the required table movement and / or C-arm translation has been calculated, but the rotation / turning of the C-arm has not been calculated. In this case, there may then be an optional additional step where the C-arm angle is adjusted as needed for the expected next imaging at the target ROI_2. This can be done by the user. Still in an alternative, some imaging geometry configuration changes, although calculated, are not automatically implemented, such as the rotation / turning of the X-arm. The adjustment is still done manually by the user through the user interface IG-UI, but the system provides machine assistance, for example, through tactile, visual (video or image) or other signals / stimuli as described above in the various semi-automatic embodiments of the system SYS.

[0130] Now for reference Figure 5B The flowchart above illustrates a computerized approach for training machine learning models (MLCs), particularly as shown in the diagram. Figure 4A , 4B As explained in the text.

[0131] At step S610, training data is received in the form of pairs including images representing a specific region of interest and associated geometric changes of one or more other regions of interest.

[0132] At step S620, images and non-image data, optionally including patient biometrics represented in the images and / or image settings, are applied to the initialized machine learning model M to produce output.

[0133] The deviation of the output from the associated changes in imaging geometry is expressed via a cost function. f To quantify. In step S630, one or more parameters of the model are adjusted in one or more iterations within the inner loop, for example, to reduce residuals.

[0134] The training method then returns to step S610, where the next pair of training data is fed into the outer loop. In step S620, the aggregated residuals of all pairs considered up to that point are minimized.

[0135] More generally, the parameters of model M are adjusted to improve the objective function into a cost function or utility function.

[0136] Other embodiments The above references vectors representing points in phase space. p k The imaging geometry configuration and its changes are described. This space is typically a high-dimensional space and depends on the degrees of freedom provided by the imaging apparatus under consideration and / or the number of imaging geometry components used.

[0137] While a machine learning model M can be trained to consider the entire imaging geometry configuration across all phase spaces, it may sometimes be desirable to limit the estimation to certain subspaces of the phase space. Other imaging geometry configuration parameters outside these subspaces are not estimated. The relevant imaging geometry configuration components are manually adjusted by the user. More specifically, in embodiments involving C-arm or U-arm imagers, changes in the geometry of interest are limited to table movement / tilt along its edges in one or both of the spatial dimensions X and Y. Alternatively, changes in angular values, i.e., rotation and / or swivel of the C-arm of interest, are also possible. Furthermore, the same principles can be applied to other imaging modalities, therefore the principles disclosed herein are not necessarily associated with X-ray or C-arm-based imagers.

[0138] In addition to the purely spatial aspects of altering the imaging geometry, the dimensionality of the phase space can be enhanced by attaching the imaging setup EPX parameters to points adjacent to the phase space. Machine learning models can then operate within this enhanced generalized phase space. The imaging setup EPX can also be described by one or more vectors, each entry representing a specific setup. The imaging setup involves querying non-spatial properties of the imaging signal, such as the energy of an X-ray or ultrasound beam, or any imaging modality under consideration.

[0139] Specifically, referring to X-ray imaging, the image settings EXP specifically include the tube voltage and / or tube current of the X-ray tube, or other general settings. Other image settings EPX envisioned herein include frame rate settings and image processing parameters such as filtering and denoising, and operating modes such as digital subtraction angiography (DSA) or single-exposure imaging.

[0140] Including imaging settings during learning allows the aforementioned processing system to not only further estimate changes in imaging geometry configuration but also changes in the EPX of the imaging settings. However, in other embodiments, the settings are manually adjusted by the user, and it is only a portion of the imaging geometry configuration that is estimated and, if desired, automatically implemented.

[0141] As mentioned above, all flags or ROIs can be considered by a single machine learning component, which is a neural network (CNN). Given the current ROI, the estimated imaging geometric configuration parameter vector... k The coordinates are then encoded to encode the corresponding imaging geometry configuration changes with respect to some or each of the other ROIs. However, in other embodiments, multiple different machine learning models are formed, with one machine learning model dedicated to each specific flag. In other words, the model can be parameterized by the corresponding flag ROI.

[0142] Alternatively, multiple different machine learning models can be provided specifically for different current image geometries. For example, one model can be used to work with all or a subset of ROIs imaged in the front-to-back imaging direction, while another model can be used to reuse ROIs imaged in the side-view direction.

[0143] Alternatively or concurrently, in embodiments, multiple machine learning models can be parameterized by aspects of their own imaging geometry configuration. In other words, changes in imaging geometry configuration for a specific subspace and phase space are considered to be parameterized by values ​​of imaging geometry configuration parameters in different phase spaces. More specifically, multiple values ​​can be used, for example, to predict table motion parameterized on images with different C-arm rotations or slews. In these models, the user first selects the corresponding rotation or slew of the C-arm and then selects the corresponding machine learning model, which then predicts the desired table motion in one or both spatial dimensions X and Y.

[0144] In another, more simplified embodiment with constrained estimation, (one or more) MLCs may only estimate the virtual... Figure 2 The subset of imaging geometric configuration changes associated with sub-segments of the marker locations in the model. Then, additional marker locations in the model are obtained computationally cheaper, for example, through interpolation or extrapolation of nearby markers predicted by one or more MLCs. The interpolation or extrapolation process can be driven by point-by-point registration of the predicted marker locations with the corresponding model markers.

[0145] In addition to image settings such as EPX settings, these can be adjusted automatically or based on user confirmation via a suitable input user interface. Adjustments can be based on patient pose estimation for the current or next imaging geometry configuration.

[0146] While the above embodiments allow the user to select the next ROI given the current ROI, a more refined embodiment is envisioned where the navigation system SYS itself fully predicts the next ROI based on the currently used imaging protocol, given the current ROI. In other words, it not only predicts imaging changes but also which ROI might be considered next. The user can override this ROI estimate or confirm it via an input user interface, responding to which the corresponding imaging geometry configuration change is estimated.

[0147] In all the foregoing, the imaging geometry configuration change and the estimation of the next region of interest can be used to achieve the actual imaging geometry configuration change by controlling the corresponding actuators. The imaging geometry configuration change can be achieved completely automatically without any user intervention. Alternatively and preferably, a graphical or other user interface allows the user to confirm the estimation, and only then is the imaging geometry configuration change achieved through actuator control.

[0148] In this embodiment, during the imaging process, accessed flags are stored or “cached” in memory. This cache allows the user to quickly switch back between previously accessed regions of interest without using machine learning components. This allows for savings in time and energy because computation via MLC is not required.

[0149] As previously described, the machine learning component encodes model M for the distribution of poses of different landmarks across the object population. Occasionally, the imaging geometry configuration estimate may not be entirely accurate. This can happen, for example, if a given patient does not perfectly fit the previously considered population sample. In such cases, the system's user interface (preferably graphical) supports the ability for the user to provide feedback on the actual pose of landmarks (such as the center of the heart) compared to the estimated pose (and therefore implicitly, the pose) of the changed imaging geometry configuration. This corrective feedback can be used, for example, to improve the estimation for subsequent estimates by adding, as an offset, the difference between the confirmed pose (e.g., position) and the previously estimated position.

[0150] While the imaging system described above has been assumed to include an actuator AC with an encoder, non-encoded X-ray systems are also within the scope of this disclosure. Non-encoded actuator systems use motor-powered or manually operated actuators that do not track their corresponding position or rotation. Such non-encoded actuators can be found, for example, in mobile X-ray systems. For this type of imaging device with non-encoded actuators, the proposed system SYS can be trained to predict classifications rather than regressing to imaging geometry configuration change vectors. p k In other words, in these embodiments, the machine learning model M can be trained to estimate qualitative directional instructions for motion and thus estimate classifications such as "right," "left," "up," "down," etc. However, even in non-coding systems, regression methods can still be used, where the model M is trained to regress the input data into orientation vectors (such as (unit) vectors representing orientation (e.g., for the movement of the inspection table TB)) to achieve changes in imaging geometry. The unit vector can define the axis of rotation and the direction of rotation.

[0151] Qualitative spatial instructions or direction vectors are preferably indicated to the user via display, audio, video, or a combination thereof. For example, with video assistance, the user can then operate actuators to achieve corresponding movements or rotations of the imaging geometry components involved.

[0152] Although the aforementioned system can operate without acquiring additional interventional fluorescence fluoroscopic images during imaging geometry configuration changes, such acquisition remains beneficial for non-coded X-ray systems in conjunction with corresponding update direction commands until the imaging geometry configuration change is complete.

[0153] The above principles can be applied to both X-ray and non-X-ray imaging devices, but they are particularly useful for X-ray-guided interventional procedures, including PCI, neurovascular intervention, and peripheral vascular intervention.

[0154] The applicant has observed that the estimated geometrical changes demonstrate significant accuracy across the board. While optional biometric non-image data... a k While it can be used with input images in embodiments, it has been found that fully trained systems deliver available estimates for changes in imaging geometry solely based on images. Therefore, it appears that image information (i.e., local information) in a particular image already globally encodes the appropriate spatial relationships for the patient to accurately estimate changes in imaging geometry. In other words, local spatial information in the image regarding the spatial relationships between local anatomical structures is clearly sufficient to reliably estimate global spatial relationships for organs or regions of interest outside the current field of view. Even the relatively small field of view used in interventional C-arm procedures is sufficient for the system to be satisfactorily trained. Similarly, if a larger field of view is used, the system's performance can be expected to improve.

[0155] One or more features disclosed herein may be configured or implemented as / have circuitry encoded in a computer-readable medium, and / or combinations thereof. Circuitry may include discrete and / or integrated circuits, application-specific integrated circuits (ASICs), programmable FPGAs (Field-Programmable Gate Arrays), system-on-a-chip (SoCs), and combinations thereof, machines, computer systems, processors and memories, and computer programs.

[0156] In another exemplary embodiment of the invention, a computer program or computer program unit is provided, characterized in that it is adapted to run on a suitable system the method steps of one of the methods described in the foregoing embodiments.

[0157] Therefore, a computer program unit can be stored on a computer unit, which may also be part of an embodiment of the present invention. This computing unit can be adapted to perform or cause the execution of the steps of the methods described above. Furthermore, it can be adapted to operate components of the apparatus described above. The computing unit is adaptable to automatic operation and / or to execute user commands. The computer program can be loaded into the working memory of a data processor. Therefore, the data processor can be equipped to perform the methods of the present invention.

[0158] This exemplary embodiment of the invention covers both computer programs that initially use the invention and computer programs that, through updates, transform existing programs into programs that use the invention.

[0159] Furthermore, the computer program unit may be able to provide all the necessary steps to perform the process of an exemplary embodiment of the method as described above.

[0160] According to yet another exemplary embodiment of the present invention, a computer-readable medium, such as a CD-ROM, is provided, wherein the computer-readable medium has computer program units stored thereon, the computer program units being described in the preceding sections.

[0161] Computer programs may be stored and / or distributed on suitable media (particularly, but not necessarily, non-transient media), such as optical storage media or solid-state media provided together with or as part of other hardware, but the computer programs may also be distributed in other forms, such as via the Internet or other wired or wireless communication systems.

[0162] However, computer programs can also be provided on networks such as the World Wide Web and can be downloaded from such networks to the working memory of a data processor. According to yet another exemplary embodiment of the invention, a medium is provided for making computer program units available for download, said computer program units being arranged to perform a method according to one of the embodiments described above.

[0163] It should be noted that embodiments of the invention have been described with reference to different subjects. In particular, some embodiments are described with reference to claims of the method type, while other embodiments are described with reference to claims of the device type. However, unless otherwise stated, those skilled in the art will understand from the above and below description that any combination of features relating to different subjects, except for any combination of features belonging to one type of subject, is also considered to be disclosed in this application. However, it is possible to combine all features to provide more synergistic effects than a simple sum of features.

[0164] Although the invention has been illustrated and described in detail in the accompanying drawings and the foregoing description, these illustrations and descriptions should be considered illustrative or exemplary, and not restrictive. The invention is not limited to the disclosed embodiments. Those skilled in the art, through studying the drawings, the disclosure, and the dependent claims, will be able to understand and implement other variations of the disclosed embodiments in practicing the claimed invention.

[0165] In the claims, the word "comprising" does not exclude other elements or steps, and the words "a" or "an" do not exclude multiple. A single processor or other unit can perform the functions of several items recited in the claims. Although specific measures are recited in dissimilar dependent claims, this does not indicate that combinations of these measures cannot be advantageously used. No reference numerals in the claims should be construed as limiting the scope.

Claims

1. A system (SYS) for supporting imaging operations of an imaging apparatus (IA) capable of presenting different imaging geometries, comprising: An input interface (IN) is used to receive the current image of the current region of interest (ROI_1) acquired by the imaging device (IA) under the current imaging geometry configuration (p), wherein the imaging geometry configuration includes the relative orientation of the frame of the imaging device and the inspection table; A pre-trained machine learning component (MLC) is configured to compute output data representing an imaging geometry configuration change (Δp) for a next imaging geometry configuration (p') relating to a next region of interest (ROI_2), wherein the next imaging geometry configuration includes a change in the relative orientation of the gantry of the imaging apparatus and the inspection table, the change in relative orientation including a change in the orientation of at least one of the gantry of the imaging apparatus and the inspection table; and The output interface (OUT) is used to output the technical parameters of the changes in the imaging geometry configuration.

2. The system of claim 1, further comprising a control module (ISC) that instructs one or more actuators based on the technical parameters to cause the imaging device to present the next imaging geometry configuration.

3. The system according to claim 1 or 2, comprising a user interface (UI) configured to allow a user to specify the next region of interest (ROI_2).

4. The system according to claim 1 or 2, comprising a graphics generator (GG) for generating a visual indication of the technical parameters on a display device (DD).

5. The system according to claim 1 or 2, wherein, The imaging geometry configuration includes the orientation of the frame (G) of the imaging device and / or the orientation of the inspection table.

6. The system according to claim 1 or 2, wherein the machine learning component (MLC) is further configured to calculate the image acquisition settings of the imaging device associated with the next region of interest and / or the imaging operation.

7. The system according to claim 1 or 2, wherein, The machine learning component includes a neural network.

8. The system according to claim 7, wherein, The neural network in question is a convolutional neural network.

9. The system of claim 1, wherein the machine learning component (MLC) is further configured to train a machine learning model for supporting the imaging operation, and the machine learning component (MLC) is further configured to: Receive training input data, which includes training images representing regions of interest (ROI_1) and associated imaging geometry configuration changes related to one or more other regions of interest (ROI_2); The training data is applied to the machine learning model to obtain training output data; and The parameters of the machine learning model are adjusted based on the training output data.

10. The system according to claim 9, wherein, The machine learning model has a neural network architecture.

11. A method for supporting imaging operations of an imaging device capable of presenting different imaging geometry configurations, comprising the following steps: Receive (S520) the current image of the current region of interest (ROI_1) acquired by the imaging device (IA) under the current imaging geometry configuration (p), wherein the imaging geometry configuration includes the relative orientation of the frame of the imaging device and the inspection table; Output data is computed (S530) based on a pre-trained machine learning component (MLC), the output data representing an imaging geometry configuration change (Δp) for a next imaging geometry configuration (p') related to a next region of interest (ROI_2), wherein the next imaging geometry configuration includes a change in the relative orientation of the gantry of the imaging apparatus and the inspection table, the change in relative orientation including a change in the orientation of at least one of the gantry of the imaging apparatus and the inspection table; and Output (S540) the technical parameters of the changed imaging geometry configuration.

12. The method of claim 11, further comprising training a machine learning model to support the imaging operation, including the following steps: Receive (S610) training input data, the training input data including training images representing regions of interest (ROI_1) and associated imaging geometry configuration changes related to one or more other regions of interest (ROI_2); The training data is applied (S620) to the machine learning model to obtain training output data; and The parameters of the machine learning model are adjusted (S630) based on the training output data.

13. An image processing apparatus, comprising: The system (SYS) according to any one of claims 1-10; as well as The imaging device (IA).

14. A computer program product comprising a computer program, said computer program being adapted, when run by at least one processing unit (PU), to cause said processing unit (PU) to perform the method according to claim 11 or 12.

15. A computer-readable medium storing a computer program or a pre-trained machine learning component as used in a system according to any one of claims 1-10, wherein the computer program, when run by at least one processing unit (PU), is adapted to cause the processing unit (PU) to perform the method according to claim 11 or 12.

Images