Adaptive Collimation for Interventional Radiography

Abstract

A system SYS and associated methods for facilitating collimator adjustment in radiography or radiation therapy. The system comprises an input interface IN for receiving input data including i) an input image, and / or ii) user input data including partial collimator settings for a collimator COL of an X-ray imaging device IA. A collimator setting estimator CSE of the system calculates complementary collimator settings for the collimator based on the input data. Preferably, the system uses machine learning.

Description

[Technical field]

[0001] The present invention relates to systems and methods for facilitating collimator adjustment in x-ray imaging or radiation therapy, methods for training machine learning models for use in such systems, computer program elements, and computer readable media. [Background technology]

[0002] Certain medical interventions, such as percutaneous coronary intervention (PCI), may require a clinician to introduce one or more medical devices or tools, such as guidewires or catheters, into a patient.

[0003] The intervention may be performed under radiographic guidance, where one or more images, which may be a series of images forming a video feed, are acquired by an imaging device ("imager") and displayed in real time to a clinician (hereinafter "user") on a display device.

[0004] The displayed image allows the user to study a lesion, an organ, an introduced device / instrument, or generally a region of interest ("ROI"). To obtain the best diagnostic or therapeutic outcome, the ROI needs to be visualized in a proper pose, which requires adjusting the imaging geometry of the imager.

[0005] During the course of these sometimes difficult and demanding interventions, the user may have to switch from one ROI to another, which may require readjusting the imaging geometry of the imager. For example, the user may first treat a stenosis in one branch of a coronary artery, then switch to another branch where a bifurcation needs to be treated. Or the treatment protocol may require switching from one organ to an entirely different organ in an entirely different part of the body. Adjustments to the imaging geometry, especially when required multiple times, are tedious and time-consuming. For example, table or C-arm movements are time-consuming, increasing the overall procedure time. Often, x-rays continue to be taken during table movements to monitor the current FOV and allow the operator to determine when the desired imaging ROI has been reached. Thus, the total user and patient radiation dose incurred during the procedure increases.

[0006] The change in imaging geometry may include adjusting the collimator of the imager. A collimator is a device that allows one to limit the shape and / or size of the imager's line beam, thus allowing the imager's field of view ("FOV") to be aligned over the ROI.

[0007] Collimation of the x-ray beam is important for both patient dose and image quality. Studies have shown that routinely collimating to the minimum required field of view significantly reduces patient and user radiation dose. Changing collimator settings is often tedious and time consuming.

[0008] Most x-ray systems require the user to manually perform collimation. Currently, collimation requires multiple user interaction steps to set up the collimation. In some imaging systems, there may be six or more different collimator components (e.g., four shutters and two wedges) that require positioning and angle adjustment. The user may have to interact with multiple physical actuators for each component, or multiple elements on a touch screen.

[0009] If the user takes the time to optimally adjust the collimator, it may be necessary to readjust the collimator after any movement of the device, gantry, or patient table / column being imaged. Naturally, the more aggressive or "tight" the user sets the initial collimation setting, the less likely it will be possible to subsequently use that setting for another imaging geometry, since it is more likely that the region of interest will no longer be within the FOV. One example of this is during radial access, where it is useful to collimate the device as it moves up the arm and into the body. Summary of the Invention

[0010] Therefore, there may be a need for an imaging assistance system that addresses at least one or more of the above-mentioned shortcomings in current imaging systems.

[0011] The object of the present invention is solved by the subject matter of the independent claims, further embodiments are incorporated in the dependent claims. It is noted that the aspects of the invention described below also apply to the associated methods, computer program elements and computer readable media.

[0012] According to a first aspect, there is provided a system for facilitating adjustment of a collimator in x-ray imaging or radiation therapy, the system comprising: an input interface for receiving input data including user input data including a first collimator setting for a collimator of the x-ray imaging device; and a collimator setting estimator that calculates a second collimator setting for the collimator based on the input data.

[0013] In an embodiment, the input data further includes at least one input image acquired by an X-ray imaging device. The input image may be a projection X-ray image in the current field of view of the imaging device. The input image may be acquired at a lower dose than the image acquired after the collimator settings have been calculated and applied. In the proposed system, fluoroscopic X-ray imaging is preferably used. Although only one such input image is sufficient, a series of input images for a given projection direction can also be used. The series of images may include an initial uncollimated image and a series of images that were collimated at an earlier point in time. User input data may be specified within such input images, for example a collimator line may be specified in the image by the user. The lines and the image may be processed together by the estimator to calculate a second collimator setting. Providing the first collimator setting together with the image makes the process more robust.

[0014] The first collimator setting may be partial or incomplete, while the second collimator setting completes or complements the first partial collimator setting. Thus, the first and second collimator settings together may form a complete collimator setting that can be used to control the collimator to achieve a user's desired collimation.

[0015] In an embodiment, the collimator setting estimator is implemented as a trained machine learning model, in some such embodiments, the second or complete collimator setting is regressed in the final layer of the machine learning model.

[0016] Alternatively, the collimator setting estimator calculates the second / complementary collimator setting based on output data provided by the trained machine learning model. In some such alternative embodiments, the output data is not provided by the final layer, but is provided as an intermediate internal output by a hidden layer of the model. That is, in such embodiments, the output data generated by the machine learning model includes a feature map. The calculation of the feature map may include the use of a nonlinear activation function, as opposed to an output obtainable at a final output layer of the model, where a nonlinear activation function is not necessarily used or where an activation function is used in a different manner.

[0017] In an embodiment, the machine learning model is an artificial neural network.

[0018] Feature maps include derived feature maps such as activation maps or heat maps for classification or regression tasks.

[0019] In general, the information captured in a feature map represents the weight activations (e.g., convolution filters) of a particular layer. Optionally, the feature map may be visualized. Although feature maps may be represented in a data structure similar to the (input) image, feature maps are a different data type than images. In images, contrast is given by variations in intensity that represent how the object being imaged (the patient or part thereof) interacts with the imaging signal, such as the interaction of tissue with x-rays in radiography. Images are image domain data, whereas feature maps are feature domain data. Variations in feature maps represent how parts of a machine learning model act on features in the input image and / or on higher level features abstracted from the image domain. For example, feature maps represent what relevance (e.g., in terms of weights, etc.) is given by a machine learning model / algorithm to a given image feature in a given hierarchy. In general, feature maps have a higher dimensionality than the input image from which they are computed by the ML model / algorithm. Feature maps are data generated by machine learning models. Specifically, feature maps are generally multi-dimensional data generated by applying convolution operations to the input. Feature maps may be suitable for extracting or abstracting features from the image domain to higher dimensions at multiple levels / hierarchies, abstracted from the image domain to, for example, “features of features” or “features of features of features” and similarly as one progresses to higher hierarchies depending on the depth of the machine learning model used. The feature maps may be dimensionally reduced to 1D, 2D, or 3D, etc., and may be optionally color coded to obtain derived feature maps, also referred to herein as heat maps. Thus, heat maps are post-processed feature maps. Post-processing to perform dimensionality reduction of the feature maps may include weighted summation, such as gradient-based methods, or simple flattening / resampling, such as global pooling methods. Heat maps can be used to globally represent how the machine learning model operates / interprets input data, either in training or deployment.

[0020] In an embodiment, the model is an encoder-decoder model, such as an autoencoder. An autoencoder is a special case of an encoder-decoder, where the original input is reconstructed at the output, i.e. the input space is the same as the output space. An encoder-decoder is more general and can decode the encoded data to any other desired output.

[0021] In an embodiment, the collimator setting estimator includes a segmenter that segments the feature or heat map into at least one segment, and the calculated complementary or second collimator setting is further based on the segment. The segmenter working with an encoder-decoder network preferably operates on the feature map, whereas in the case of an autoencoder network, the segmenter preferably operates on the heat map. In an autoencoder network, feature maps of any hidden layer can be used for segmentation. In a general encoder-decoder network, it is preferable to use the feature map of the final layer. However, it is also envisaged to access feature maps from previous layers. The segmenter may operate on color parameters in a color space, such as hue, saturation, etc., if the feature map / heat map is colour coded.

[0022] The segmenter itself may be implemented as an ML model, or may be configured as a traditional analysis processing path such as SIFT.

[0023] Having the segmenter operate on feature maps or heatmaps in feature space makes it more robust to unpredictable variations that may occur in real-world situations compared to operating on regressed images. If the final output data were used instead of feature maps, such segmentations would likely be highly specific to a given data type. However, in feature space, segmentations are discovered automatically and unsupervised, making them more robust to variations in the data. Thus, feature maps can be used more generically.

[0024] End-to-end ML segmentation computation may require ground truth data, which is time-consuming and costly. In the proposed embodiment, segmentation is performed in feature space, which is simpler. Since feature maps can be color-coded, segmentation can be based on color characteristics (hue, saturation, etc.). Thus, segmentation in feature space is preferred herein, although in some embodiments segmentation in image space is not excluded here.

[0025] Because the nonlinear activation function maps to a range of values ​​rather than a binary set, edges in the feature map or in the segmentation based on the heat map may be smoothed. The output of the activation function represented by the feature map contains smooth transitions that can be interpreted as collimator settings, potentially providing information about the tightness of the collimator.

[0026] The user input or additional input provided by the user may be based on feature maps, segmented feature maps, or heat maps. Selecting a value close to or toward one end of the range of values ​​of the activation function may be interpreted as moderate collimation, while selecting a value close to the other end of the range may represent a more conservative, i.e., tighter, collimator preference. In other words, the output of a smooth activation function provided by some machine learning models (deep learning) is utilized here for segmentation. The raw output (without thresholding) of the activation function is preferably used for segmentation.

[0027] In an embodiment, the first / partial collimator setting comprises the specification of a geometric curve or line within an input image, or activation map, acquired by an imaging device.

[0028] The user input may include one or more points, one or more parallel lines, diagonal lines, or curved lines. This information is projected into the collimator setting space by the proposed system and method. The user input, or optional additional user input, may include indicating geometric elements (points, lines, curved lines) in a feature map, a segmented feature map, or a heat map. The user input may be scalar and within the range of the activation function, and therefore may be interpreted as indicating a collimator tightness setting and processed into a collimator tightness setting.

[0029] In an embodiment, the input data further includes an input image acquired by an X-ray imaging device. Such an input image may represent a current field of view of the imaging device. Such processing of the input image is preferably envisaged herein in a feature map or heat map based embodiment, in which a machine learning model generates a feature map or heat map based on processing of the input image. The feature map or heat map is then segmented, optionally based on a user input designating one or more elements in the segmented feature map or heat map as a first or partial collimator setting. Based on the segmentation and this user input (or additional user input), a supplemented or second collimator setting is calculated.

[0030] In an embodiment, the second collimator setting may be calculated as a contour of the feature map or heat map. The tightness of the collimation may be adjusted by adjusting the threshold of the activation function.

[0031] The collimator setting is defined by one or more parameters such as collimator lines. A second collimator setting is also envisioned herein, which is an improved, refined, and more specific collimator setting than the first collimator setting. Thus, the first collimator setting can be considered as a rough estimate, rather than an incomplete one as envisioned in some embodiments above. In an embodiment, the first and / or second collimator setting parameters specify the collimation tightness.

[0032] In an embodiment, the system comprises a user input device for capturing user input data. The user input device may include any one or more of a graphical user interface, an eye tracking device, a gesture detection device, and a voice processor.

[0033] In an embodiment, the imaging device can assume multiple different imaging geometries, and the collimator setting estimator adjusts the interpolated collimator settings in response to the imaging device changing its imaging geometry, allowing a user to quickly set frequent and / or tight collimation during an imaging procedure / intervention.

[0034] In an embodiment, the second collimator setting parameter specifies the collimation tightness.

[0035] In an embodiment, the collimator setting estimator adjusts the tightness of the current interpolated collimator settings based on receiving updated partial collimator settings at the input interface.

[0036] In another aspect, there is provided a method of training a machine learning model of a system according to any of the above embodiments.

[0037] In an embodiment, the method is unsupervised, however supervised schemes are not excluded herefrom.

[0038] According to another aspect, there is provided a method for facilitating collimator adjustment in x-ray imaging or radiation therapy, the method comprising: - receiving user data including a first (e.g. partial) collimator setting for a collimator of an X-ray imaging device; - calculating a second (eg, complementary) collimator setting for the collimator based on the input data.

[0039] The supplemented collimator setting complements the initial partial collimator setting and together forms a supplemented or complete collimator setting.

[0040] The proposed system allows for a quick collimator setting. Since no complete parameter specification is required, busy medical users can save time and effort. The system utilizes the provided user data and / or input images to estimate the user's preferences and calculates the supplemented collimator setting parameters accordingly. Thus, the very same system can accommodate the individual use preferences of multiple users. The calculated supplemented collimator settings (parameters) include the user provided settings (first settings) and additional (second) settings calculated by the system, which preferably together form the intended complete collimator setting. The complete collimator setting allows for the adjustment of the collimator used, preferably uniquely. The calculated collimator settings can be used to operate the collimator symmetrically or asymmetrically.

[0041] In particular, a collimator tightness preference by the user may be derived by the system based on at least the provided first / partial collimator settings, the settings may be defined by collimator setting parameters, and the complemented collimator setting parameters are calculated taking into account the tightness preference.

[0042] The proposed ML-based configuration is generic and not application specific. It does not depend on recognizing any particular features, anatomical structures / organs, or devices inside the patient's body. The system described herein can work for any clinical application (organ, device, etc.) since the features used to set the collimation are not necessarily explicitly trained beforehand. This is because it has been found that user-provided inputs such as one or more collimator lines already provide enough context for the ML system to properly predict the user's intended collimation, especially in combination with the input image. Furthermore, the "user-in-the-loop" aspect of the system allows for improved specificity of the collimation while significantly reducing the burden on the user.

[0043] Although the facilitation of collimation described above is envisaged herein primarily for X-ray imaging, this does not exclude other applications, such as radiation therapy, where collimation to the contours of a lesion in tissue is required, in this and other applications, collimation may be performed in 3D rather than 2D as in X-ray projection imaging.

[0044] "Imaging Geometry": In fluoroscopy or other x-ray or non-x-ray imaging modalities, this may include one or more (any combination) or all of the following: angle, rotation angle, translation, etc. In general, imaging geometry refers to settings or configurations that change the position or orientation of the optical axis, or a virtual axis connecting the x-ray source and x-ray detector, and / or settings / configurations of the imaging device / equipment that otherwise spatially affect the imaging FOV relative to the imaged subject (e.g., collimator settings).

[0045] In this specification, the term "user" refers to a person who operates an imaging device.

[0046] The "patient / object" may be a human being, an animal or plant, a microorganism (eg, in microscopic imaging), etc., but may also relate to inanimate "objects", such as in baggage screening or non-destructive material screening.

[0047] In general, a "machine learning component" is a computerized configuration that implements or facilitates the implementation of a "machine learning" ("ML") algorithm. The machine learning model may be based on an ML "model". The ML component is configured to perform a task. In an ML algorithm, assuming that the training data has a suitable distribution, the performance of the task improves significantly when (new) training data is used for training. The model is adjusted based on the training data. Performance may be measured by objective testing when the trained model is fed with test data. Performance may be defined by requiring a certain error rate to be achieved for given test data. See "Machine Learning" by T.M. Mitchell, page 2, section 1.1, McGraw-Hill, 1997. The main objective of this specification is to predict collimator setting parameters, either implicitly or explicitly. [Brief description of the drawings]

[0048] Exemplary embodiments of the invention are described with reference to the following drawings, which are not to scale. [Figure 1] FIG. 1 is a schematic block diagram of an imaging device. [Diagram 2] FIG. 2 shows a perspective view of a collimator system of an X-ray imaging device and aspects of a projected image that can be obtained by collimated X-ray exposure. [Diagram 3] FIG. 3 illustrates a schematic block diagram of a computer-implemented system for facilitating adjustment of a collimator. [Figure 4] FIG. 4 shows an operational diagram of the system of FIG. 3 in an embodiment. [Diagram 5] FIG. 5 illustrates a machine learning model architecture that may be used to implement the system of FIG. [Figure 6] FIG. 6 illustrates another embodiment of an encoder-decoder type machine learning model that can be used to implement the system of FIG. [Figure 7] FIG. 7 shows the results obtained by a system using the machine learning model of FIG. [Figure 8] FIG. 8 illustrates another embodiment of an autoencoder type machine learning model that can be used by the system of FIG. [Figure 9] FIG. 9 shows a block diagram of the processing steps for the machine learning model of FIG. [Figure 10] FIG. 10 illustrates a process flow for a machine learning model-based implementation of the system of FIG. [Figure 11] FIG. 11 shows the results obtained with the system of FIG. [Figure 12] FIG. 12 illustrates a flow chart of a method for facilitating adjustment of collimator settings. [Figure 13] FIG. 13 illustrates a computer-implemented training system for training a machine learning model that facilitates collimator adjustment. [Figure 14] FIG. 14 is a flowchart of a method for training a machine learning model to facilitate collimator adjustment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0049] Referring to FIG. 1, there is shown a schematic diagram of a configuration AR for image-based assistance, preferably for use in connection with a medical intervention.

[0050] The configuration AR is an X-ray image I of the patient's internal structures in the ROI. t The ROI may be the human heart, lungs, or another organ or group of organs.

[0051] Image I i , also referred to herein as a sequence of frames, may be displayed to the user in real time on the display device DD as a moving image or video feed, or may be displayed alone as still images if desired.

[0052] The imaging arrangement AR further comprises a user assistance system USYS configured to assist a user in adjusting the collimator settings.

[0053] As mentioned above, the imaging device AI and system USYS are primarily contemplated in the embodiments herein to assist in medical interventions such as percutaneous coronary intervention (PCI). Other medical interventions are also contemplated, not necessarily performed in relation to the human or animal heart, as well as non-medical applications. Such non-medical applications may include image-based assistance to inspections or work performed in inaccessible caverns or piping systems, or inspection of technical equipment. Such technical equipment may include engines and other complex machines that cannot be directly inspected by the naked eye and require imaging equipment to provide access to occluded areas of interest so that they can be visually inspected via video feeds or still images.

[0054] Referring first to the imaging device IA in more detail, it may be configured as a C-arm or U-arm type imaging device, as shown in the exemplary embodiment of FIG. 1. In the embodiment of FIG. 1, the C-arm system IA is mounted to a ceiling CL, but this need not be the case in all embodiments. Alternatively, the imaging device IA may be floor-mounted or stand-mounted, etc. As a further example, the imaging device may be mobile, such as on wheels or mounted on a track. The gantry G is not required, for example, in a mobile x-ray device where there is no or no permanent physical connection between the detector D and the tube XX.

[0055] The X-ray imaging device includes an X-ray detector D and an X-ray source XS. In typical, but not necessarily all, embodiments, the imaging device has a gantry G that holds the X-ray detector D and the X-ray source XS, such as an X-ray tube. The X-ray detector D and the X-ray source XS are arranged in an opposing spatial relationship on the gantry G to form an examination region between the X-ray source and the X-ray detector. A patient PAT is positioned within this examination region such that the region of interest is located approximately at the isocenter of the IS imaging device. The patient may lie on an examination table TB during imaging. The table TB may be adjustable in height H, translatable along the X-axis or Y-axis, or both the X-axis and the Y-axis, and in some embodiments tiltable about one or more tilt axes.

[0056] During an imaging procedure, the X-ray source XS is activated by applying a cathode current and voltage between the anode and the cathode to generate an X-ray beam XB that emanates from a focal spot at the anode. The beam leaves the X-ray source, passes through the examination region and thus through the patient's tissue at or around the region of interest, and impinges on the X-ray sensitive surface of the X-ray detector D. The X-ray sensitive surface of the detector D may include a number of pixels that convert the impinging X-rays into intensity values. The intensity values ​​may vary from location to location, and this variation is caused by differences in attenuation of the X-ray beam due to different tissue types having locally different material densities.

[0057] The intensity values ​​so recorded by the detector XS may be mapped to image values ​​according to a color or gray value palette to form a projection image (or "frame"). The acquisition circuitry thus operates to capture a sequence of multiple different projection images at multiple different time points during the imaging procedure at an appropriate frame rate. An exemplary frame rate contemplated herein is 20-30 fps. For example, in fluoroscopy, the primary modality contemplated herein, the intensity values ​​may be mapped to values ​​ranging from black to gray values ​​to white. The darker the image value, the lower the intensity value. Other mapping schemes may be used, such as inverse mapping, where low intensity values ​​are mapped to bright image values, as commonly used in radiography. Other mapping schemes may be used instead. This specification does not exclude the acquisition of a single or a small number of still images, such as occurs in radiography.

[0058] The spatial width of the (primary) X-ray beam defines the FoV of the imager IA. The presence or extension of an object within the field of view, i.e. within the X-ray beam, changes the intensity of the X-rays locally detected at the detector. The field of view can be changed by user request or automatically by adjusting the imaging geometry of the imager IA, e.g. by moving the X-ray source, by moving the patient, by widening or narrowing the beam width using the collimator COL or all or any part of the above elements.

[0059] The X-ray detector may be configured as a digital flat panel detector communicatively coupled to a display device DD. The flat panel detector D may be of the direct or indirect conversion type. In another embodiment, the image detector may be configured as an image intensifier coupled to the display device via a video camera.

[0060] The contrast-imparting mechanism for projected images primarily envisaged herein is attenuation, although other imaging techniques that additionally or alternatively utilize other contrast mechanisms are not excluded herefrom, such as phase contrast and / or dark field imaging, etc. In the latter two cases, the imaging device may include additional components, such as an interferometer.

[0061] The imaging device includes a control console CC through which a user can determine when to start and stop the imaging procedure, and in particular when to activate the X-ray source XS. Pedals may be coupled to the console as a user interface for controlling the activation or deactivation of the X-ray source or for manipulating a grid switch to stop or resume exposure to the X-ray beam.

[0062] The main direction of propagation of the primary X-ray beam (excluding scattered radiation) is defined by the optical axis OX, which is an imaginary line extending from the focal spot of the X-ray source (not shown) to the central part of the X-ray sensitive surface of the X-ray detector D. The optical axis defines the spatial projection direction.

[0063] To better assist the user in navigation, the position or spatial orientation of the optical axis, i.e. the projection direction, can be changed upon user request. In one embodiment, this can be achieved by configuring the gantry to be rotatable about one axis, or preferably about two mutually perpendicular axes. Two such rotation axes allow two degrees of freedom for the change of the optical axis. For example, in one geometry, one rotation axis extends into the plane of the paper in FIG. 1 and allows the optical axis to be rotated by an angle β. The other rotation axis is parallel to the plane of the paper in FIG. 1 and can be oriented by another angle α, independent of β, as shown diagrammatically in FIG. 1. In general, the axis α defines the "rotation" and the axis β defines the "angle".

[0064] Optionally, the height of the gantry itself may also be variable, as indicated by the double arrow H in FIG. 1. Also, the optical axis OX may be translated by moving the gantry along a line. The position and orientation of the optical axis may be referred to herein to at least partially define the imaging geometry. In other words, the imaging device contemplated herein in embodiments allows a user to change the imaging geometry.

[0065] Changes in imaging geometry may be requested by the user manipulating a joystick or other suitable user interface UID', which may be coupled to or incorporated into the control console CC.

[0066] The request to change the imaging geometry may include applying a control signal to an appropriate actuator AC located on the imaging device, e.g., one or more of the gantry, table TB, collimator COL, etc. In response to the control signal, the actuator AC changes the imaging geometry. The actuators AC may be powered by a power source or manually by a user via a handle, lever, or other device. The actuators AC may be purely automatic, or hybrid, or semi-automatic. In the semi-automatic case, the user operates a user interface UID', such as a joystick or other control device, but may be assisted by servo motors or the like to perform the change in imaging geometry.

[0067] The actuators may or may not be encoded. If encoded, the actuators may include linear or angular encoders, such as potentiometers. The encoders allow changes in image geometry performed by one or more actuators to be tracked, i.e., mapped to numerical coordinates that change in response to changes in imaging geometry.

[0068] Other options for changing the imaging geometry may include changing the distance between the detector and the X-ray source and / or changing the distance between the region of interest and the X-ray detector and thus the X-ray source. The latter change may be performed by changing the height H' of the examination table TB on which the patient lies. Changing the height h and / or the distance between the source and the detector may result in a rescaling of the image with a certain magnification.

[0069] Yet another option for changing the imaging geometry may include translating the patient table TB in X and Y directions in a plane parallel to the surface of the table, one direction parallel to the plane of the paper in FIG. 1 and the other direction extending into the paper. The table may be tiltable about one or more axes. Components of the imaging device involved in changing the imaging geometry may be generally referred to herein as imaging geometry components, and may include, among others, any one, more or all of the following: the source XS, the gantry, the detector, the collimator COL, the table, etc.

[0070] Generally, a change in imaging geometry changes the spatial relationship of the x-ray source and / or detector to the region of interest. Additionally or alternatively, the field of view may be altered by movement of the collimator or by moving the patient, for example by translation of the table TB as described above.

[0071] Describing now in more detail the user assistance system USYS, it is configured to assist a user in adjusting the collimator settings of the collimator COL of the imaging device IA.

[0072] In general, for collimation adjustment, the user assistance system USYS works with a user input device UID. The user device UID may include a graphical user interface (GUI). The GUI may be displayed on a user monitor UD, which may or may not be different from the display DD used to display the acquired images. Such a GUI-based user input device UID may include one or more touch screen interfaces (TSNs). The GUI may include one or more graphic components visualized on the user display device UD. The graphical components may include widgets such as stylized buttons that allow the user to interact and adjust the collimator by touch screen manipulation or use of a pointer tool. The pointer tool may be a computer mouse or stylus, etc. The user input device UID may support gesture recognition. A more conventional keyboard-based user input is also envisaged, where one or more text boxes are populated with text or alphanumeric input.

[0073] Alternatively or in addition to a GUI-based embodiment, the user input device UID for collimator adjustment may (further) include a control panel with one or more sets of physical control elements, such as levers (e.g., joysticks), buttons, or other manually operable actuators. Such user interfaces with manually operable physical control elements are sometimes referred to as TSOs. In yet other embodiments, the user interface device UID may further include other interface options, such as eye tracking devices, gesture tracking systems, etc., in addition to or instead of the above. User interface devices with voice capture and interpretation capabilities are also envisioned to enable voice command-based control of the user. In an embodiment, the user control device UID may be incorporated into an augmented reality AR system.

[0074] The collimator adjustment user device UID may be at least partially integrated into an operator console unit CC, which is communicatively coupled to actuators AC of the imaging apparatus IA or other hardware or software components of the imaging apparatus IA by wireless or wired communication channels.

[0075] Briefly, as described above, the user assistance system USYS envisaged herein can communicate with the collimator adjustment user interface device UID. Specifically, the user assistance system USYS receives a collimator setting request provided by a user via the user interface device UID. At least a portion of the collimator setting can be defined by one or more parameters, referred to herein as collimator setting parameters l. In the following, it is assumed that there are multiple collimator setting parameters l, but this is not intended to limit the present disclosure, and settings that can be defined by a single such parameter are not excluded from the present specification and are specifically envisaged in the embodiments of the present specification.

[0076] The collimator setting parameters l provided by the user are processed by the user assistance system USYS to calculate a complemented, i.e. additional or improved, collimator parameter setting. More specifically, the user may only need to partially define the collimator setting parameters, with the user assistance system USYS calculating the complete, or at least complemented, collimator setting parameters l'. The system USYS:

number

[0077] The interpolated collimator settings l' may be displayed on the user display device UD or any other display device (e.g., device DD), optionally simultaneously with the current image acquired by the imaging device IA. For example, the interpolated collimator settings l' may be displayed as an overlay graphic element superimposed on the current image. Additionally or alternatively, the interpolated collimator setting parameters l' may be automatically converted by a suitable middleware into control signals / commands and forwarded via a communication channel to the actuators A-C of the imaging device to instruct the actuators to perform the requested collimator settings at the collimator COL. Alternatively, the calculated interpolated collimator setting parameters l' may be displayed with a user interface option for the user to approve the calculated parameters. The control signals / commands are sent only upon approval by the user, such as by touch screen operation or other user input.

[0078] Thus, the user assistance system USYS allows the user to save time and effort. Instead of the user spending effort and time to specify a complete set of collimator setting parameters, the system USYS assists the user in this task. The user only needs to provide a possibly small portion of the collimator setting parameters l, and the proposed system USYS provides the necessary collimator setting parameters l. * Thus, users, experienced or not, may be able to adjust the collimator settings more quickly. In some embodiments, some sort of dynamic loop control subsystem may predict the remaining collimator setting parameters l as soon as a new user-defined collimator setting l is provided. *The system can be considered a closed loop ("in-loop") where the user is part of the loop. The ability to rapidly adjust the collimator can be beneficial in real-time interventions, especially when stress factors are high, such as in trauma environments, where the imaging geometry (and therefore the field of view) may have to be changed frequently.

[0079] With reference to FIG. 2, the collimator system COL will now be explained in more detail.

[0080] FIG. 2A) shows a perspective view of the collimator system COL with other components of the imaging device IA partially cut away for ease of illustration. In general, the collimator system COL is configured to modify the X-ray beam XB emanating from the source XS. Specifically, the collimator system COL modifies the shape and / or size of this original beam to define the width of the field of view of the imaging device. In other words, the collimator determines how much is "visible" in the acquired image given the current set of remaining imaging geometry settings, such as the pose of the table and C-arm.

[0081] More specifically, the collimator system COL is operable to change the shape and size of the cross-section of the X-ray beam in a plane perpendicular to the current orientation of the optical axis OX. Such beam modifications may include limiting the cross-section to allow for a reduction in the X-ray dose for the patient and the user, and to allow for a more focused image to be obtained in which only the current region / anatomy of interest (ROI) is displayed. The collimator COL allows for the adjustment of the cross-section of the X-ray beam by placing one or more of a plurality of radiopaque collimator blades BL at least partially within the beam XB. The blades BL, which may be formed, for example, from lead, tungsten, or other high-Z materials, are referred to herein simply as "blades BL."

[0082] The blades BL are preferably motorized by one or more actuators AC to define a field of view FOV of a desired shape and / or size. The movement of the blades allows the shape and size of the FOV to be dynamically changed as required by the user and / or protocol. The blades BL may be rotatable along one or more axes. However, in some embodiments, the movement of the blades controlled by the actuators AC is limited to translation in a plane X,Y perpendicular to the optical axis OX, as shown in FIG. 2A). As shown in FIG. 2A, multiple sets of such blades may be provided in a separated state along the optical axis OX. One set of blades (in FIG. 2A, four blades are shown, but there may be more or fewer) may also be referred to as shutters, and the other set of blades closer to the X-ray source XS along the imaging axis OX may be referred to herein as "wedges". Generally, no distinction is made herein between shutters and wedges, and both are referred to herein simply as "blades BL" unless otherwise specified.

[0083] FIG. 2B) is a plan view along the optical axis OX (running perpendicular to the plane of the paper in FIG. 2B) of the collimated field of view FOV. The shadow or radiation footprint of the blades BL is shown as Π(BL). In FIG. 2B), as in FIG. 2A), four blades BL are shown, some or all of which are movable towards the optical axis in order to restrict the field of view from one, several or all sides. Thus, the FOV size is reduced, but can be expanded again (to the original cross section of the beam XB at a given magnification) by retracting some or all of the collimator blades, depending on the user's wishes. Instead of symmetric collimation (as shown in FIG. 2B), asymmetric collimation is also envisaged herein, in which only a subset of the collimator blades are activated and moved into the beam XB. In asymmetric collimation, some or each blade BL can be moved independently of one, some or all of the other blades.

[0084] In the projection image of FIG. 2B, the projection of the edge of the collimator blade close to the axis OX is represented as a set of four lines, one for each collimator blade, which together define the field of view. These lines are referred to herein as "collimator lines". Geometrically, each of these lines can be described by two parameters ax+b. A system with up to N such geometric lines in the projection plane of the detector can be used as a geometric device to define the desired collimator setting, where N is the number of blades. More specifically, the collimator lines themselves may represent one way of describing the collimator setting parameters. For this reason, the symbol "l" is used to refer to both in this disclosure. However, other descriptions of collimator setting parameters are also envisioned, such as descriptions in terms of control commands and / or blade position / rotation (length dimension and / or angle measured relative to a reference pose), user controller position (e.g., joystick position), etc. The exact nature of the collimator setting parameters that the user can specify / request may depend on the type of user input device UID. However, regardless of the nature of the collimator setting parameters, they can always be considered equivalent to a set of collimator lines as described above. Thus, in this specification, even when referring to collimator setting parameters / collimator settings in general, we refer to such collimator lines. This is not a limitation of the disclosure herein, as other ways of specifying collimator setting parameters are also envisioned. Such other collimator setting parameter formats are not necessarily converted to such lines in all embodiments. However, specification by the user via a user input device UID of such lines, particularly if GUI-based, is specifically envisioned in embodiments herein. In other embodiments, if the specification is not of such lines, appropriate middleware can be used to convert the parameters to a set of collimator lines and then provided as input to the system USYS. In other embodiments, native processing of other formats is also envisioned.

[0085] FIG. 2C) shows a detailed view of a collimated field of view to explain the concept of tightness of collimation. Tightness of collimation refers to the clearance or distance h that remains between the collimator lines and the structure or region of interest ROI, as indicated by shading in FIG. 2C). The smaller the clearance h of at least a portion of the region of interest, the stronger the collimation. In extreme cases, at least one of the collimator lines forms a tangent to the region of interest captured within the geometric structure in the image recorded by the detector D.

[0086] The plane of the drawings in Fig. 2B) and C) is the plane of the detector D. Thus, Fig. 2B) can be interpreted as a part of the actual projection image acquired by the detector in that plane. Given the remaining imaging geometry settings, especially the magnification (distance between the X-ray source XS and the detector plane D), specifying a line in the detector plane can uniquely define the collimator settings of one or all of the collimator blades. That is, given a current set of imaging geometry settings, the calculated interpolated collimator settings, when provided as a set of lines, can be converted by the middleware into a corresponding set of commands and used to drive the actuators AC to execute the calculated collimator settings l'.

[0087] Thus, in the embodiments herein, it is envisaged that the user can simply indicate a single (or multiple) collimator line or a portion thereof on the touch screen, for example by specifying a single or multiple parameters in any vector (x,y) or parameter representation. For example, an angle representation or the like may be used. For example, the user selects a single geometric point on the image. The orientation of the collimation line may be perpendicular or parallel to one of the predefined image axes. In another example, the user provides two geometric points that are collinear with the collimation line. In yet another example, the user provides a single geometric point, and the initially indicated orientation is adjusted by changing the orientation angle. Alternatively, for example, the user is specified by performing a line drawing gesture. The touch screen may display the current image or other representation of the current FOV. The (geometric) line thus specified may constitute a partial collimator setting l. This may be processed by the proposed system USYS to calculate the remainder for the complement of the other three detector lines, etc., to provide a complete set of collimator setting parameters l'. The complete set can be used to drive the actuators to perform the corresponding collimation.

[0088] In addition to or instead of simply specifying one collimator line or a portion of that line (above-mentioned portion), the user may specify at least a portion of the region of interest in a rough outline. These two items of information, i.e., the geometric region of interest and the at least one collimator line, may be used by the system USYS in an embodiment to calculate a complete set of parameters l′ that can be used for the complete collimator adjustment.

[0089] The completed collimator settings can be displayed graphically on the display device DD, UD as an overlay graphic on the current image. Only the calculated completed parameters l* can be displayed as such, or the combined set

number

[0090] As mentioned above, when specifying the region of interest as an optional parameter, it may not be necessary for the user to completely surround the region of interest. This is because, in an embodiment, the current image is pre-segmented by the system USYS into a multi-segment structure, and the user only needs to identify the desired segment representing the desired ROI with as little as one point, such as by touch screen manipulation. At least a partial ROI specification, together with a partial collimator line or a portion thereof, is sufficient to calculate the complemented collimator setting parameter l'. More importantly, as proposed herein, instead of specifying the region of interest and the collimator line or a portion thereof, a clearance estimate δ between the two is defined. This clearance estimate may be interpreted by the system, in an embodiment, as an indication of the desired collimation tightness h. The estimated collimation tightness is automatically applied to the remaining collimator lines to define the complete collimator setting l' with the required tightness h. In another embodiment, instead of specifying the collimator line or a portion thereof, the user simply specifies a measure of the collimator tightness, for example a normalized value between 0 and 1. Based on this collimation tightness measure, the system USYS calculates the complete collimator setting parameter l'. In an embodiment, the collimation tightness parameter δ is estimated from the user input and the predicted segmentation mask or predicted heat map. For example, h can be estimated by calculating the perpendicular distance between the collimator line and the nearest feature with the highest activation value (e.g., value 1 in the segmentation mask, see e.g., FIG. 6) (in other words, h=0). Collimator tightness is described in more detail in FIG. 6.

[0091] Thus, the proposed system USYS is not only able to calculate the complete collimator setting l', but also to detect the user's preferences, i.e. the desired collimation tightness, which is implicit in the user input provided. This further allows to facilitate the operation of the imaging device IA, since the required collimation tightness depends on the user's preferences and may therefore vary from user to user. This may be important in a clinical environment where multiple users use a given imaging device IA. Also, during a given medical procedure, the required collimation tightness may change. Parts of the procedure (e.g., an intervention) may require very tight collimation, while other parts of the procedure may be more suitable with a looser collimation scheme.

[0092] Also, as proposed herein in some embodiments, the collimator settings are dynamically adjusted in real time in response to changes in other aspects of the imaging geometry of the imager IA. For example, an event handler (not shown) built into the system USYS intercepts control commands directed to angulation and other aspects of the imaging geometry, such as changes in table position / height, as described above. When the event handler intercepts such data indicating an intended imaging geometry change, this data is forwarded to the system USYS and processed such that the current collimator settings are adjusted to accommodate the new field of view as a result of the imaging geometry change. The intercepted data is likely generated by a user interface element of the console CC. For example, such a signal may be generated by a joystick or other UI element that is operated by the user to request a change other than the collimation imaging geometry. Preferably, in this embodiment, the same collimator tightness associated with the current collimation settings is maintained and applied to the new FOV, and no further user input regarding collimation parameters is required. In another example, the collimator tightness changes with the imaging geometry (either field of view angle, magnification, etc.) to account for expected apparent drift of the anatomical structures of interest. If this fully automatically proposed new collimation setting does not suit the user's preferences, the user can again provide partial user input as described above. Thus, the system USYS calculates the new / updated missing collimator setting parameter parts l* and calculates the full set of collimator setting parameters l' for the new FOV. The full set of parameters is then automatically applied or first displayed, if necessary.

[0093] Reference is now made to the block diagram of Fig. 3, which shows in more detail the proposed user system USYS. As proposed in the present invention in a preferred embodiment, the system USYS is implemented by using machine learning. In particular, the system may comprise a machine learning model M, which is trained or pre-trained on training data, as will be explained in more detail below.

[0094] Now, assuming that the system has been sufficiently trained and its parameters tuned, during the post-training phase, deployment, one or more input interfaces IN of the system USYS receive user inputs u supplied by a user via a user interface device UID.

[0095] The user input u may include only a partial specification of the intended collimator setting parameters l, as described above. The user input u may include one or more line sections of a collimation line. A plurality of collimation lines may be included in the user input u (but preferably not all). Additionally or alternatively, no collimation lines are provided, but other parameters suitable for specifying the intended collimator setting are provided. For example, the collimator setting parameters provided by the user may include a quantification of the collimator tightness clearance δ, as described above with reference to FIG. 2C). Optionally, the currently acquired X-ray projection image I0 recorded by the detector D in the current detector plane is also received at the interface IN. As described above, the user input may further include a partial or complete specification of the intended ROI that the user wishes to collimate. The user input may include one or more collimation positions or shutter positions generated via an existing interface (TSM or TSO) or other interfaces (gestures, eye tracking, voice, AR, etc.).

[0096] Next, user input u, preferably including the current image I0, is processed by the collimator setting estimator CSE to calculate the completed or improved collimator setting from the provided input data. To do this, the collimator setting estimator CSE includes a trained machine learning model M to which some or all of the input data u is applied. In some embodiments, the intended completed or improved collimator setting parameters l' are calculated end-to-end by the machine learning model M itself. In other words, all calculations are performed by the machine learning model M. However, this is not the case in all embodiments, as will become more clear when describing other embodiments below.

[0097] As will be explained in more detail below, the machine learning model M can be configured as an artificial neural network of various architectures. A preferred type of architecture includes computational nodes arranged in a series of layers. In an end-to-end embodiment, the intended output is regressed to a regression result. The regression result represents the intended perfected improved estimated collimator setting parameters l'. In particular, the regression result is provided by the final layer of this model M. The user input is processed by the machine learning model to extract the user's preferences for the intended collimation settings, embedded within the provided user input and / or the current input image I0.

[0098] However, such an ML-based end-to-end embodiment is not required in all embodiments. Specifically, in other embodiments, machine learning is used as well, but the internal data generated in the hidden layer of the machine learning model is processed by other components of the estimator CSE to obtain the results. Instead of using the data provided as the final output at the output layer, as in the end-to-end ML embodiment, in the non-end-to-end ML embodiment, it is the intermediate data that is utilized. This internal intermediate data may include feature maps. Feature maps represent the more latent abstract patterns that the model was able to extract from the input data. By their nature, the internal feature maps are generated by a given hidden layer as the output of the activation function that is part of the given hidden layer. Feature maps may represent the distribution of various features in image regions in the feature space as an abstraction of the image content.

[0099] In some non-end-to-end ML-based embodiments, the feature map is segmented. In particular, it has been found that the feature map is more robust to certain individual idiosyncrasies of the input data or image I0. Thus, in embodiments, this feature map is processed by a computational component different from the model M to obtain the interpolated collimator setting l'. In embodiments, the feature map may be reformatted, e.g., resampled to correspond in size to the current image I0. For example, the feature map φ, whether resampled or not, may be processed in embodiments by the analytical image segmenter stage SEG. The image segmenter SEG does not necessarily use machine learning methods. More specifically, in embodiments, no prior training data is used to train the segmenter SEG. However, ML-based embodiments of the segmenter SEG are also envisioned. Some feature maps may be color-coded by hue, saturation, or brightness, and the color-coded feature maps are processed and optionally displayed. Non-ML type segmenters contemplated herein include feature-based detection such as SIFT or other transforms, or simple threshold-based segmentation, segmentation using color space (hue or saturation, pixel intensity), region growing algorithm-based segmentation, etc. By analyzing or segmenting the feature map to calculate missing collimator setting parameters, the intended user's preferences regarding the desired collimator setting can be better captured. Some embodiments of the ML-based segmenter SEG contemplated herein include deep learning segmenters SED, such as those in the U-Net architecture, Mask R-CNN, and variants thereof, and other artificial neural networks, preferably fully convolutional. If color encoding is used, the segmenter SEG operating on the feature map or heat map may use color / hue / saturation rather than using edges or pixel intensity as in other types of segmentation (embodiments that do not use color encoding are also contemplated herein).

[0100] More broadly, the machine learning model is a transformation involving three different spaces: i) an input space, which may include the image space represented by the current image I0, ii) a feature space, and iii) a collimator parameter space. In an end-to-end ML embodiment, the result is calculated by the transformation from the input space to the collimator parameter space via the feature space. In a non-end-to-end ML embodiment, ML is used only to perform the transformation to the feature space. In such a non-end-to-end embodiment, the feature map in the feature space is accessed by an additional computational component, such as a segmenter SEG. The accessed feature map is processed by the additional computational component into the sought collimator parameters in the collimator parameter space.

[0101] In both end-to-end ML and non-end-to-end ML embodiments, the calculated collimator setting parameters l' are output at the output interface OUT. As described above, the output results l' can be used to automatically drive the actuators of the collimator to apply the desired estimated collimator settings. Additionally or alternatively, for example, a graphic rendering of the collimator line system can be displayed on the display device UD, DD. The visualizer VZ component may generate a graphic overlay in which the resulting graphic rendering is superimposed on the current image I0. In yet other embodiments, the collimator setting parameters may be stored in a database to provide an event log for regulation, or the collimator setting parameters may be processed in another way.

[0102] Referring now to FIG. 4, there is shown the operation of the proposed system at different time points t0, t1 and t2 during an intervention where images are acquired by an imaging device with a changeable imaging geometry.

[0103] At an initial time t0, a current frame I0 is acquired. The user specifies a partial collimator setting parameter l, for example by touch screen manipulation. For example, the user may specify one of four possible lines,

number

number

number

[0104] Further, optionally, as shown in pane C, the collimator settings may include settings for wedges, shown with diagonal lines in pane C, while panes A and B refer to shutters. At a later time t1>t0, the user may decide to change one (single) or multiple parameters of the current collimator setting parameters. This is shown in pane D, where the collimator lines previously specified by the user are changed.

number

[0105] In response to a change to one (or more) of the current collimator settings, the remaining collimator parameters are adjusted accordingly, as shown in pane E. If multiple lines are being changed, this can be done simultaneously or sequentially.

[0106] Pane F shows the situation at a later time t2>t1, where the current imaging geometry has been changed, for example by increasing the magnification or reorienting the gantry. In response to a request for a change other than the collimator imaging geometry, the collimator settings are adjusted to dynamically track the region of interest, preferably without additional user input. If further user input regarding the collimator is then received, the collimator tightness and / or remaining collimator setting parameters are recalculated as in panes A and B.

[0107] Reference is now made to Fig. 5, which shows a block diagram of a machine learning model M that may be used by the system USYS in an embodiment. Fig. 5 shows an embodiment of an end-to-end ML variant. In particular, Fig. 5 shows a block diagram of a machine learning model in an architecture of the artificial neural network type.

[0108] Preferably, a convolutional network CNN is envisaged. The network may be fully convolutional or partially convolutional and may include one or more more fully connected layers. Alternatively, the entire network may be fully connected. Some or all layers may be recurrent. Additional recurrent layers may be included. The left side of FIG. 5 shows the user input u, including optional region a and partial collimator setting parameters l. The network may be feedforward as shown in the figure, but may alternatively be recurrent.

[0109] User input may be provided through a GUI. A current image I0 or a series of images is displayed and the user specifies one or more (but preferably not all) intended collimator lines. In addition to or instead of specifying the collimator lines, the collimator tightness h is selected by the user, for example by specifying a scalar value within a predefined range. One end of the range indicates tight / tangential collimation, the other end of the range indicates more permissive collimation at a given maximum distance from the ROI. Optionally, as described above, the intended region of interest a is specified by outlining it or simply indicating one or more points that define the region of interest. The collimator lines preferably intersect or at least touch the ROI. In this way, the displayed lines have a dual function, since they indicate the collimation lines and at least a part of the ROI. The region of interest may be specified on a single input image or on a series of images (frames). Such a series of frames, i.e. a video feed, may be generated in a fluoroscopic imaging operation, as specifically envisaged in the embodiments herein.

[0110] The input data u is provided in an appropriate format, such as a vector or matrix format. The input data preferably includes an input image and one or more collimator lines specified in the input image, but may also include pointer tool or touch screen actions, etc. The input data is received and processed in an input layer IL of the neural network M. The outputs of the input layer are then passed sequentially through the network. The data is then passed through one or more hidden layers L1 to L N and finally regressed by the output layer OL to a regression result R, which represents the estimated interpolated collimator setting parameters. The output is a system of coefficients (a i , b i For example, six sets of coefficients (a i , b i ) (i=1..6), the collimator lines for four shutters and two wedges can be defined. a' is the x-intercept and b' is the y-intercept. The collimator lines for blades without rotation capabilities, such as wedges, may be described by a single coefficient b. If the blades are rotatable, a reference orientation may be defined. For example, the orientation may be assumed to be orthogonal to one of the image axes. The regression result thus includes the imputed collimator setting parameters. In particular, the imputed parameters include one or more lines that were missing in the incomplete input data. On the right side of FIG. 5, the imputed line is shown diagrammatically as the predicted line l'. Thus, FIG. 5 shows an example of a deep neural network that predicts other collimation setting parameters given an image ROI extracted from input parameters (e.g., user interaction with one collimation variable). In the example of FIG. 5, the collimation line on the right side is used to predict the collimation line on the top side.

[0111] Input layer and hidden layer OL, L1~L NPreferably, x contains a nonlinear activation function such as ReLu(x), arctan(x), or sigmoid. The output layer OL, on the other hand, may or may not implement such a nonlinear activation function. However, since the image size is known, it is advantageous to use such a nonlinear activation function to restrict the output at layer OL to a finite, known interval, e.g., the unit interval. The slope of the activation function determines the smoothness (further below. In contrast to activation functions such as ReLU in the hidden layer, the activation function in the output layer determines the task that the network will perform (e.g., regression, etc.).

[0112] The feature map φ is the vector of data from network L1 to L N . Preferably, there are no feature maps generated by the output layer. At some or each hidden layer, one or more feature maps are computed. The feature maps may be computed by sending the respective logits to an activation function of that hidden layer or of another hidden layer. The logits are the result of processing the feature maps of the previous layer by a convolution operator, or other operator depending on the configuration of a given hidden layer. An activation function is applied to the logits of this hidden layer to compute a feature map of that hidden layer. This feature map is passed to the next layer, and so on. The previous layer is either another hidden layer or an input layer. In this way, a product of a sequence of feature maps is generated, the product increasing with the depth of the layer. In general, the number of feature maps per layer also increases with depth, since a given layer may implement multiple operations, e.g. convolutions that differ from each other, and the number generally increases with the depth of the layer.

[0113] One or more feature maps in a given hidden layer may be processed by a segmenter SEG or other computational component, as described above in non-end-to-end ML embodiments, as will be described immediately below.

[0114] In particular, a deep convolutional neural network is preferably used. The depth depends mainly on the number of hidden layers, or the number of passes through the hidden layers in a recurrent network. The network M in FIG. 5 may include one or more intermediate / hidden layers. The number of hidden layers, and therefore the depth, may depend on the complexity of the application or the availability of training data.

[0115] Each layer may include one or more operators, such as convolution operators, batch normalization operators, dropout, and pooling. Each hidden layer, and in some embodiments the input layer, includes an operator for implementing a nonlinear activation function (preferably not the output layer). As data passes through the network, a multidimensional feature map, or a series of feature maps, is generated by the hidden layers. The output layer OL converts one or more feature maps into a single or a series of low-dimensional embeddings. Global average pooling or map flattening techniques are used in the output layer, followed by a recurrent layer. As mentioned above, a sigmoid activation function or other nonlinear activation function may be applied to the output from the recurrent layer to constrain the output parameters.

[0116] Typically, there are multiple feature maps at a given depth and / or across the depth of the network. Preferably, the last layer (before the output layer OL) is accessed by the segmenter SEG for segmentation as described above. However, alternatively, feature maps of any hidden layer may be accessed. Feature maps located deeper in the architecture M ("distal" from the input layer IL) are preferred, since the more distal the layer, the more abstract or complex the underlying task structure being encoded.

[0117] Reference is now made to FIG. 6, which illustrates a block diagram of a machine learning model M for an embodiment of a non-end-to-end ML implementation. In this embodiment, a single image I or multiple X-ray images are input u=I j, and is propagated through multiple layers. As mentioned above, the feature maps are generated in a hidden layer and / or an output layer OL. The output layer OL may be a recurrent layer. In this embodiment, the feature maps are processed separately (by a segmenter SEG or other entity) outside the machine learning model M for which they are generated. This is in contrast to end-to-end ML embodiments in which the feature maps are not so processed, but instead are processed all the way to the output layer. In the (final) output layer OL, multiple feature maps are combined with the regression results, for example in model M of FIG. 5. This type of feature map-based processing can also be implemented in the network shown in FIG. 5, but in a preferred embodiment, an autoencoder ("AE") type network is used, as shown diagrammatically in FIG. 8. In another embodiment, an encoder-decoder (EC-DC) type network may be used, as shown diagrammatically in FIG. 6. In FIGS. 6 and 8, the rotated triangle symbol

number

number

[0118] We now first elaborate on the Encoder-Decoder type of network. Everything stated above for the network of FIG. 5 applies equally, except that the output from the network of FIG. 6 is a feature map instead of a set of regression parameters. In the encoder-decoder configuration, the input image is first converted to a lower dimensional representation x by the encoder component EC of the encoder / decoder network. This lower dimensional and / or smaller size representation is referred to herein as a "code". The code x is then upsampled by the decoder DC component arranged in series with the encoder EC. This increases the dimensionality / size of the central representation, and ultimately the dimensions and / or size of the output image match those of the input image. Thus, the "code" "encodes" the characteristics or structure of the input image into a compact form such that it can be reduced and still be decoded back to the input image. The term "size / dimension" refers to the number of entries in the feature map or input / output image data structure and / or the vector length of the set of indices to locate such entries. For example, a matrix (e.g., an image) of 50x50 pixels has a larger size than an image of 40x40 or 50x40. However, both have the same dimensions. The three-channel image 50x50x3 has a larger dimension (3D) than the 50x50 2D image. The 3D image has more data points and therefore a larger size. Also envisaged is an encoder-decoder embodiment in which the encoder EC generates a code of larger size and / or higher dimensionality than the input image to foster an over-complete representation. In this alternative embodiment, the encoder EC may generate a code of larger dimension and / or size, but with greater sparsity. In particular, regularization mechanisms and special cost functions may be used to promote over- and sparse codes. The encoder part may include multiple hidden layers, each implementing a convolution operator and preferably also an activation function applied to the output of the convolution.The stride or step width of the convolution operator may be greater than 1 to achieve a reduction in size / dimension of the feature map. The decoder DC may include multiple hidden layers implementing operators to increase the size / dimension. A transposed convolution operator may be used, which essentially acts as the inverse operation of the convolution caused by the convolution operator in the encoder EC. While the convolution operator initially maps from pixels to increasingly higher level features, the transposed convolution operator maps features to pixels and / or lower level features. The hidden layers of the decoder DC may also implement activation functions. Functionally, the deconvolution operator may be formulated in terms of convolution operations that are then summed. See, for example, section 2 of "Adaptive Deconvolutional Networks for Mid and High Level Future Learning" by MD Zeiler et al., 2011 International Conference on Computer Vision, Barcelona, ​​Spain.

[0119] Here, to explain the feature maps in more detail, they are b, which may be scaled in size to correspond to the size of . It can be seen that the smooth boundary parts of the segmentations in the feature maps are obtained thanks to the activation function output used to compute the feature maps. This smooth boundary effect is due to the class of activation functions envisaged herein that may map smoothly, albeit nonlinearly, to a limited range, such as the unit interval [0,1], which is not configured for the hard thresholds typically used in some segmentation CNN architectures. Some such smooth activation functions have an S-shaped profile with a slope approaching 0 towards both ends of the range and a maximum slope, e.g., about 1, at a given location in the range (e.g., 0.5). This given location is referred to herein as the threshold of the activation function. Thus, in the feature maps, the values ​​vary from 0 to 1, locally defining the smooth boundary parts of the natural segmentations encoded in the feature maps. These segmentations may emerge in a given feature map during the processing of the input data by the network M. The segmentations in the feature maps represent the above learned latent patterns in the input data. The network may contain multiple feature maps per hidden layer, and there may be multiple such hidden layers. Preferably, the last feature map from the decoder DC is used, or alternatively, any feature map in any hidden layer in the decoder DC path may be used.

[0120] In an encoder-decoder type embodiment such as in Fig. 6, the output from the last layer of the decoder stage DC is preferably used as a feature map for segmenting the collimator setting parameters sought. However, it may not be necessary to use the feature map from the last layer of the decoder stage DC, since feature maps from upstream layers before the decoder stage DC may be used after resizing to match the input dimensions. Those previous (or subsequent) feature maps are preferably in the last decoder layer, k=3 layers.

[0121] The smooth boundary portion in the segmentation example is shown as δ in Fig. 6. As shown on the right side of Fig. 6, the smooth boundary portion is used in this embodiment to calculate the complementary collimator setting parameter l * As in the previously described embodiment, the user can still specify one or more collimator lines, for example in the segmented feature map, but this is not necessary. Instead, a desired collimation tightness metric is specified by the user for one segmentation segment in the feature map, for example in the last feature map from the decoder path.

[0122] Specifically, in the boundary portion δ, the user only needs to define a point. This then corresponds to the output value of the activation function, which is used by the estimator CSE as an anchor point for constructing the isoline in the feature map. The isoline directly defines the collimator curve of the intended collimator setting. The collimator curve may be divided into a system of approximation lines to obtain the system of straight collimator lines described above. However, the division into lines is not necessarily required, such as in the case of multi-leaf collimators used in radiation therapy. With such multi-leaf collimators, the collimator curve may define a FOV shape that is better approximated than a straight line. Thus, the specification and / or output format of the calculated and interpolated collimator setting parameters envisaged herein is not limited to lines, but may be any curve, such as a high-order polynomial. Piecewise defined systems of linear and nonlinear curves are also envisaged in the embodiments herein. The isoline embodiment may be advantageously used in radiation delivery devices such as the Cyberknife® system, where collimation to diseased tissue of complex shape may be required.

[0123] Instead of selecting points at the boundaries of the feature map segments, as described above, the user may adjust the threshold of the activation function itself, such as by a slider or other suitable user interface, touch screen, etc. Each adjustment recalculates the feature map, producing boundaries with different levels of smoothness. By segmenting the feature map for edges of segments with adjusted levels of boundary smoothness, the tightness of the collimation can be controlled. The segments in the feature map can be interpreted as collimation masks.

[0124] The neural network used for feature map extraction as shown in Fig. 6 is preferably a deep fully convolutional network. Any NN model suitable for segmentation tasks may be envisaged herein, such as U-Net, R-CNN, MaskR-CNN, etc. If the network has an encoder-decoder architecture, the last convolutional layer of the decoder DC preferably has a sigmoid activation function applied to the values ​​of the last feature map.

[0125] In any of the above feature map based processes, a heat map can be used instead. A heat map is a special type of feature map obtained by post-processing a feature map. The post-processing to derive a heat map may include weighted summation followed by non-linear activation. A heat map may be displayed for control purposes to highlight regions of interest. The generation of a heat map is described in more detail in Figs. 8 and 9.

[0126] It will be appreciated that the encoder-decoder configuration described in Figure 6 can also be used in combination with the machine learning model shown in Figure 5. In this embodiment, the feature maps generated by the network of Figure 6 are used as input to the network of Figure 5 instead of the input image I0. The regression output provided by the last layer OL of the network of Figure 5 generates the collimator setting parameters.

[0127] Reference is now made to FIG. 7, which shows an illustration of the results obtained by the EC-DC based ML system described above, specifically the system of FIG. 6. FIG. 7 shows selected fluoroscopic image results from a mitral clip deployment procedure. Starting from the left, each row A-C and D-F shows the positions where the collimator lines of the input image, the proposed collimation mask, the wedge (orthogonal lines) and the shutter (diagonal lines) are shown, respectively. The user can adjust the shown positions by adjusting the activation threshold, e.g. from 0 to 1, or by selecting a boundary value and calculating the collimator lines as isolines as described above.

[0128] Encoder-decoder networks are trained in a supervised manner. At each training iteration, an input image is fed into the network and the network predicts a "soft" collimation mask (the last feature map from the decoder DC path). This is then compared to the ground truth collimation mask using a predefined loss function. Examples of loss functions are mean squared error (MSE) or cross-entropy loss. The network's parameters are automatically adjusted based on the difference calculated between the ground truth collimation mask and the predicted mask. The adjusted parameters include the weights and biases of the intermediate layers. During the training process, the loss value is minimized and is stopped when a certain stopping criterion is met. The ground truth collimation mask is provided by an expert user, such as a physician.

[0129] Referring now to Figure 8, there is shown another embodiment of a non-end-to-end ML-based implementation using an autoencoder configuration similar to that described above in Figure 6. The model is preferably a deep fully convolutional network. Preferably, but not necessarily, a feed-forward architecture is used.

[0130] AE-type networks are preferred over networks like that of FIG. 6, since autoencoder-type networks can be trained unsupervised. The learning objective of an autoencoder is for the decoder to replicate the encoder input at its output to encode the underlying structure in the data, i.e. to extract a robust feature map. Unsupervised learning schemes other than AE-based are also envisaged herein. If a supervised configuration is used in some alternative embodiments, the learning objective may be any, such as classification of image objects. Preferably, the supervised learning is based on applicable images, such as X-ray projection images, that show a legitimate distribution of image objects (e.g., organs, parts of organs, or footprints of medical instruments / devices such as catheters) that can be reasonably expected to appear and that may be used during the intervention for which the proposed collimator settings are intended. As in the above case, the system is trained on input data, where the input received at the encoder stage EC is replicated at the output of the decoder DC stage. In the embodiment of FIG. 6, a single X-ray image I0 or multiple X-ray images are input u=I j is applied to the input layer as, and propagated through multiple layers of the model.

[0131] As mentioned above, the feature maps are generated in the hidden layer. The learning or training phase is based on an optimization algorithm. The optimization can be implemented as one or more iteration cycles i:=i+1, during which a cost function L is improved by updating the parameters of the AE network. The cost function L is the function of the output

number

[0132] The trained encoder EC is used during the unpacking process as follows: b is applied and feature maps are generated in the encoder as the input image passes through it. After passing through a sufficient number of layers in the encoder, the appropriate feature map φ is accessed by the estimator CSE. For example, the code x, which usually has the smallest dimension or has the highest sparsity, may be accessed as a special feature map. Alternatively, a feature map φ of an upstream layer of the encoder EC may be accessed, which is preferable, especially if heatmaps are computed from the feature maps.

[0133] In an embodiment, a feature extraction algorithm is then used by the segmenter SEG to process the feature map (e.g., φ=x) and extract features x(φ) to define segmentation segments. The feature extraction algorithm may be analytical (non-ML based) or ML based. Any feature extraction algorithm may be used.

[0134] Specifically, in an embodiment, the segmenter SEG preferably receives resampled feature maps extracted from the last convolutional layer of the encoder EC sub-network. The resampling operation is a dimensionality reduction operation, e.g., multiple feature maps or high-dimensional feature maps are reduced to 1D or 2D. Typically, resampling is applied to the feature maps to ensure that their dimensions match the input size.

[0135] In an autoencoder type embodiment such as in Fig. 6, the feature map output by the last layer of the encoder stage EC (code x) is preferably used as the feature map for segmenting the collimator setting parameters sought. As with the encoder-decoder configuration of Fig. 6, it is not necessary to use the feature map of the last layer of the encoder EC. Feature maps from earlier upstream layers may be used, as well as feature maps from the first or first few back layers of the decoder stage DC. These earlier or back feature maps are preferably in the k=3 layer of the last encoder layer of code x.

[0136] In an embodiment, instead of using feature maps, derived feature maps, referred to herein as heat maps, are used. In general, a heat map is a derived or differentiated feature map whose values ​​represent the suitability of a particular feature map of a hidden layer to the entries of the final output of the output layer OL. There are multiple approaches to extract feature maps from the feature maps of a network, all of which are contemplated herein. In some embodiments, a gradient-based approach is used to extract the heat map from the feature map. In such an embodiment, a differentiator in the CSE calculates the gradient of the latent variable (code x) output by the encoder EC with respect to one of the hidden (preferably convolutional) layers of the encoder EC part of the autoencoder. The hidden layer is preferably the last layer before the autoencoder code x, or any intermediate hidden layer in the encoder EC path. Thus, the weights calculated from the gradient-based approach for each filter output are obtained. The heat map is then extracted by weighting and summing the corresponding filter outputs, followed by applying an activation function such as ReLU to reinforce or restrict certain values. Alternatively, this may be performed without weighting the output by simply applying global average pooling or other cross-channel operations (e.g., global max pooling followed by a nonlinear activation function). The results may be normalized and scaled to the desired size of the input image if necessary. Other methods may be used, such as the Grad-CAM method by Ramprasaath R. Selvaraju et al., “Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization”, 2016, published online as preprint arXiv:1610.02391.

[0137] The segmenter SEG segments the heatmap into one or more segments. Preferably, the segmentation operation may be based on similarity. Any segmentation method known in the art may be used, such as k-means pixel clustering, region growing, active contours, simple thresholding, or machine learning methods based on U-Net, R-CNN, Mask R-CNN, etc. Another segmentation approach envisaged for the segmenter SEG in the embodiment is to calculate one or more different descriptors ds (see FIG. 10D) for a portion or each isoline of the heatmap, for example by calculating the gradient magnitude and direction at some or each point on the isoline of the heatmap, or based on the intensity values ​​of neighboring pixels.

[0138] Based on the extracted features / image segments, a collimator setting parameter l' can be calculated. This calculation is preferably interactive, i.e., based on user input l. The user individually specifies the segments themselves, as above, for example, by specifying a single (or multiple) point in the respective image segment. Additionally or alternatively, a desired collimator tightness clearing h is provided, as above. Additionally or alternatively, for a given image segment, one or more (but preferably not all) collimator lines are shown. The lines may intersect or border the image segment of interest. The lines may be set to pass through smooth boundary portions of the segmented feature map. Thus, as above, the lines not only specify the image segment, but also define the collimator tightness.

[0139] The predicted collimator setting parameters are proposed by the system using the generated segments. For example, the system USYS can calculate several candidates of the tangents of each polygon defined by the contours of the segments to calculate the complementary collimator lines. Each candidate represents a collimation setting line and is thus defined by a y-intercept (parameter b) and a slope (parameter a). For example, the tangent point is the y-intercept and the slope is defined by the gradient direction at this point. The input provided by the user, i.e. the collimation line setting, defines the collimation tightness based on the proximity to one of the candidates. The remaining collimation setting lines are selected based on their similarity to such proximal candidates using keypoint descriptors.

[0140] FIG. 10 illustrates the feature map-based process for computing collimator parameters based on a pre-trained encoder stage of an autoencoder configuration. A) An input image, e.g., the current acquired image I0, is processed by the encoder stage EC, and B) a feature map φ is generated. C) The feature map, optionally resized to match the size of the input image, is treated like a segmentable image. The feature map may contain segments b with smooth boundaries. D) Gradient magnitude and phase, or other features, may be extracted from the feature map. The feature map may be processed into a heat map, and it is the heat map, rather than the (original) feature map, that is processed from now on. The segmented segments of the feature map may be overlaid with keypoint descriptors calculated at each location on the feature map, or at each segment in the feature map extracted, e.g., using a k-means clustering algorithm or other algorithms known in the art (thresholding, region growing, active contours, color-based methods). E) The input u is provided by the user as a line (shown as a dotted line), preferably intersecting the feature map with the corresponding descriptor. The bottom rows F)-I) show user scenarios. F) For example, the user provides feedback that may define the level of collimation tightness. For example, moderate collimation tightness is represented by lines far away from the region of interest (e.g., the projected footprint of the instrument). For example, for higher tightness levels, the lines provided are closer to the footprint of the instrument. H) Wedge and I) shutter collimation lines are proposed and may be overlaid on either or both of the augmented image and the original X-ray image. The augmented image is obtained by compositing the original X-ray image with a heat map / feature map.

[0141] Figure 11 illustrates the results obtained by processing the feature maps of a neural network architecture, such as an autoencoder, as described above in Figures 8 to 10. The X-ray images represent a mitral clip implantation procedure. The top row (A,B) shows tight collimation, whereas the bottom row (C,D) shows moderate collimation.

[0142] It can be appreciated that predicting collimation parameters according to FIG. 5 may not be feasible from a small patch of pixel data (ROI designation data "a"), especially if the user input is inaccurate. Alternatively, predicting collimation parameters according to FIG. 5 may be application specific. The feature map based embodiment of FIGS. 6 to 10 addresses this. The embodiment of FIG. 6 uses a segmentation operation, preferably on the last feature map. A possible drawback of this embodiment is that it requires a ground truth collimation mask. In the embodiment of FIGS. 8 to 10, feature extraction is first applied to extract (rich) features from the feature map, and the collimator setting parameters are based on the extracted features represented as feature maps. The embodiments of FIGS. 8 to 10 do not require ground truth, and therefore can be trained unsupervised. Comparing the embodiments of FIGS. 6 and 8, they can be summarized as follows: The embodiment of FIG. 6 represents a segmentation network. The last (or other) feature map from the decoder DC is used to calculate the collimation lines. The embodiment of FIG. 8 represents a feature extraction network. Any intermediate feature layer at any hidden layer, typically in the encoder path (including the code x), can be used to compute the collimation lines.

[0143] Now referring to Fig. 12, Fig. 12 shows a flow chart of a method for facilitating collimator adjustment, preferably based on the above machine learning model. However, it should be understood that the steps described below are not necessarily tied to the above model. Instead, as described above, any end-to-end or non-end-to-end ML model can be used, and optionally, feature map-based additional processing with or without ML can be used.

[0144] In step S1210, input data is received. The input data may include a current X-ray projection image and / or data supplied by a user. The user data is supplied by a suitable user input device, such as a graphical user interface. The GUI may be touch screen based, or the like. Alternatively, the user data input data is generated by the operation of physical actuators, such as buttons, joysticks, levers, handles, or the like, preferably using encoded trackers, eye movement tracking, gesture recognition, or the like.

[0145] The input data may include a collimator tightness parameter, a single numeric value between 0 and 1, or in some other (finite) range. Additionally or alternatively, the input data preferably includes an incomplete description of a desired collimator setting. For example, the user input data may include a single or multiple collimator lines, preferably less than the total number of lines required to fully specify the collimation setting. Other input data equivalent to or corresponding to such one or more collimator lines are also contemplated herein. User input data, such as one or more collimator lines, may be specified within the input image.

[0146] In step S1220, the input data is processed using a machine learning model that has been pre-trained based on the training data. Step S1220 can be implemented end-to-end by the machine learning model. The input data is fed into the model to generate a regression result that represents the desired collimation setting, appropriately supplemented, improved, or otherwise added to, in the final layer of the machine learning model. The processing in the final layer preferably does not include processing with an activation function.

[0147] In addition, or in an alternative embodiment, rather than using the final result generated without activation in the final layer of the machine learning model, the output of the hidden layer used in step S1220 is used instead to calculate the imputed collimator setting parameter l'. The output is generated as a hidden layer feature map. The processing of the feature map may include the use of a nonlinear activation function. The activation function of a given hidden layer is applied to the logits. The logits are obtained by processing the output of the previous layer using other operations such as convolution or transposed convolution.

[0148] In this feature map-based embodiment, an analytical non-machine learning based segmentation algorithm may be used to segment the feature map or heat map. The feature map / heat map itself may be appropriately scaled to have the same size as the input image received in step S1210. Segmentation may include extracting features from the feature map. Thus, the feature map is treated and processed like an image. The segmentation operation may be feature-based, such as SIFT. The segmentation operation may obtain individual segments in the feature map.

[0149] In such feature map-based embodiments, it is the segments of the feature map (scaled or not) that are used to calculate the interpolated collimator setting parameters. In an embodiment, once the segmented feature map is calculated, (further) user input regarding the designation of one or more segments is received from the user. Based on the segments, and optionally using the user input originally provided, collimator setting parameters such as collimator lines are calculated. The collimator lines may be calculated as edges of the designated segments. The edges may be calculated to have the necessary clearance from the segments to provide the collimator settings with the required collimation tightness. Due to the activation function, the boundaries of the segments may be smooth. Further user input may include the designation of as little as one point on the smooth boundary. Thus, the interpolated collimator lines may be calculated as isolines of the boundary portion of the feature map values ​​at that point. This value may thus designate a tightness parameter, since the boundary portion generally decreases in value as one moves beyond the boundary portion and away from the segment. Thus, the value of the activation function encoded in the feature map may be utilized herein as an indication of the desired collimator tightness. The segment portion values ​​can be used as described above by constructing isolines over the feature map to find the desired segmentation. Alternatively, the collimator line of a given tightness level is obtained by the user adjusting the threshold of the activation function of the hidden layer to generate several different versions of the feature map with structures with different smoothness of the boundary transitions. Thus, segmenting such structures results in several edges that represent the desired collimator line with different tightness.

[0150] Thus, in an embodiment, the (further) input is simply a scalar value indicating the desired collimator tightness, which is then used to calculate the complete collimator setting parameters from the feature map. User input may further include the specification of the respective segments.

[0151] In step S1230, the calculated segmentation setting parameters are output and overlaid on the initial current input image received in step S1210, or the calculated interpolated collimator setting parameters are applied to the collimator blade actuators of the imaging device to achieve the desired collimation.

[0152] In step S1240, it is determined whether the current imaging geometry has been changed by the user. If so, the collimator settings are discarded and the previous steps S1220-S1230 are repeated, but with the steps based on a new image representing a new field of view acquired by the imaging device in the new imaging geometry.

[0153] In embodiments such as in Figures 6 to 10 where the input includes one or more images, this may be further enhanced by other context information of the imager system IA such as C-arm position, fluorine flavor, treatment card, treatment phase identification, device identification in the image, protocol settings, system log files, etc. This context data may be processed together with the image data. The processing may be done in separate layer strands, preferably fully combined to obtain a feature map for the context data. The feature map for the context data may be concatenated with the feature map generated during processing of the image data, potentially resulting in a multi-channel feature map.

[0154] In any of the above models in Figures 5, 6, and 8, the input u may not only include a single (current) image I0 as above, but may also include a sequence of images.

number

[0155] 13 and 14, aspects of the training are described in more detail.

[0156] Referring first to Figure 13, there is shown a training system TS for training a model M. The training operation involves learning the model parameters, i.e. the weights of the neural networks discussed in Figures 5 to 10, or the weights of other neural network type models, or the weights of non-neural network type ML models.

[0157] The training data is k sets of data (x k ,y k), where k can be in the tens, hundreds, or even thousands. For each pair k, the training data consists of training input data x k and the associated target y k Thus, the training data is organized in k pairs, especially for supervised learning schemes such as the models in Fig. 5 or Fig. 6. The model in Fig. 5 includes an image and a set of line parameters (a i , b i ) is required, whereas the model of Fig. 6 requires data pairs consisting of an image and a "soft" collimation mask, the size of which matches the size of the input image. However, unsupervised learning schemes are also envisaged here, such as the autoencoder type networks described above in Figs. 8 to 10, where the training data consists of "trivial" pairs of two identical images, e.g. (x k ,x k ), xk is image).

[0158] In supervised learning, the training input data x k is obtained from historical X-ray projection image data acquired for previous patient cases and maintained in an image repository, for example a PACS in a hospital information system (HIS). k Or "ground truth" may represent, for example, labels.

[0159] For example, a training dataset for a model M according to FIG. 5 or other supervised learning scheme may include a relatively large number of preferably (independent) different images I obtained from multiple different procedures and sampled from a relatively large patient population. kThe ground truth data may be generated from a set of training images (k=1···N>>30). The ground truth data may be generated by expert users. Each expert user looks at the training images and selects the desired collimator setting parameters. The training images may contain partial collimator settings that can be obtained by hiding the complete collimation represented in the history images. Thus, the expert users use their expert knowledge to infer the missing collimation parameters when labeling. Thus, the labeling exercise may be organized and managed as a "game". The collimator setting parameters may represent the position of the collimator blades and may be represented by a parametric equation with coefficients a, b of the collimator lines as described above. The lines may be straight or curved. Patches of pixels of any size may be extracted around each line and combined with the parameters (a,b) and stored in memory as a 2-tuple associated with the training input images.

[0160] In the encoder-decoder scheme described in Fig. 6, a collimation mask is created from collimator setting parameters provided by an expert user, for example by generating the largest possible area enclosed by intersecting collimation lines. In the autoencoder scheme described in Fig. 8, no labels are needed and the ground truth is given by the training input y k =x k is equal to

[0161] If training includes contextual data, such as in the multi-strand model described above in relation to Figure 6, then the target y for any pair k is k does not generally include context data. In other words, for learning with context data, the pair is generally ((x k ,c),y k ) where the non-image context data c is the training input x k associated only with target y k is not associated with.

[0162] In the training phase, an initial set of weights is preset for the architecture of a machine learning model M, such as the CNN network shown in Figures 5 to 10. To preset the weights, random initialization, or synthetic or independent pre-training with identically distributed training data, or a combination can be used. The weights θ of the model NN are determined by the parameterization M θ The objective of the training system TS is to obtain training data (x k ,y k ) pair. In other words, learning can be mathematically formulated as an optimization scheme in which a cost function F is minimized, but one could also use a dual formulation that maximizes a utility function instead.

[0163] We now assume the paradigm of a cost function F, which measures the aggregated residual, i.e., the error between the data estimated by the neural network model NN and the targets for some or all of the training data pairs k.

number

[0164] Specifically, the network M in FIG. 5 is pre-trained on the task of regressing input data onto the parameters of collimator lines with missing data. Thus, the regression can be one or more coefficients "a" (x-intercept) and "b" (y-intercept) that define the collimator lines in a parametric way. The network FIG. 5 can be trained using an MSE loss function F that compares the predicted lines with associated target lines that may have been selected by an expert user in the labeling task. Other cost functions may be used.

[0165] In the feature map based processing of Figures 6 to 10, the model M is preferably an autoencoder whose task is to regress the input image onto itself. If a non-autoencoder network is used for feature map based processing, any network M can be used, which can be trained on any task classification or regression, using training images that represent the most likely scenarios encountered during deployment. For example, the network M can be trained for image detection of objects on X-ray images, such as TEE probes and guidewires. In this embodiment, the task is irrelevant, since only internal feature maps from hidden layers of appropriate depth are used. In an autoencoder, the preferred feature map is the code, which is the feature map generated in the last convolutional layer of the encoder EC part. More specifically, the autoencoder type networks of Figures 6 to 10 are preferably trained using an unsupervised feature learning paradigm. It is assumed that the encoder sub-network EC learns the best fitting features, reduced to a low-dimensional latent representation, the code x, which can be decoded by the decoder DC to match the input image. The consistency between the input and the output may be optimized using either the mean squared error (MSE), mean absolute error (MAE), structural similarity (SSIM), binary cross entropy, adversarial loss, or any combination of these as the loss function F.

[0166] In training, the training input data x k is passed through the initialized network M. Specifically, the kth pair of training inputs x k is received as input IL, passed through the model, and the output training data M θ (x) is output as output OL. The appropriate measure || || implemented by one of the cost functions F above (e.g., p-norm or squared differences) is the actual training output M generated by the model M. θ (x k ) and the desired target y k The difference (also referred to herein as the residual) between

[0167] The output training data M(x k ) is the input training image data x k Target y associated with k In general, this output M(x k ) and the associated target y of the kth pair currently under consideration. k Using an optimization scheme such as backward / forward propagation or other gradient-based methods, we tune the parameters θ of the model M to fit the pair (x k ,y k ), or we can reduce the residuals on a subset of training pairs from the full training dataset.

[0168] The model parameters θ are the current pair (x k ,y k After one or more iterations in the first inner loop, where the updater UP updates the training data pair x k+1 ,y k+1 is processed. The structure of the updater UP depends on the optimization scheme used. For example, the inner loop managed by the updater UP may be implemented by one or more forward and backward passes in the forward / backward propagation algorithm. During parameter tuning, the aggregated, e.g. summed, residuals of all training pairs up to the current pair are considered to improve the objective function. The aggregated residuals may be formed by constructing the objective function F as the sum of the squared residuals of some or all of the considered residues of each pair, as in equation (1). Other algebraic combinations instead of sums of squares are also envisaged.

[0169] Optionally, one or more batch normalization operators ("BN", not shown) may be used. A batch normalization operator may be incorporated into the model M, e.g., coupled to one or more convolution operators in a layer. Using a BN operator may mitigate the vanishing gradient effect, i.e., the gradual decrease in gradient magnitude with forward and backward pass iterations that occurs during gradient-based learning algorithms in the learning phase of the model M. A batch normalization operator BN may be used in training, but may also be used in deployment.

[0170] The training system shown in Fig. 13 can be considered for all learning schemes, especially supervised schemes. Unsupervised learning schemes can also be envisaged here in alternative embodiments. The training system TS can be implemented using a GPU or a TPU.

[0171] The fully trained machine learning module M may be stored in one or more memories or databases and made available as a pre-trained machine learning model for use in the user assistance system USYS proposed herein. The trained model M may be available in a cloud service. Access may be provided free of charge or may be permitted for use on a license payment or pay-per-use basis.

[0172] Referring now to FIG. 14, there is shown a flowchart of a method for training a machine learning model in any of the above embodiments.

[0173] A suitable training data set is obtained as described in Figure 13. Preferably, a supervised learning scheme is envisaged herein, although this is not required as an unsupervised learning setting is also envisaged herein.

[0174] In supervised learning, the training data includes suitable pairs of data items, each pair including training input data and associated target training output data. The images may be obtained from historical patient records, such as a PACS or other data repository, as described above.

[0175] Continuing to refer to FIG. 14, in step S1410, the training data is divided into pairs (x k ,y k ) Each pair contains a training input x k The target y associated with k Contains: x k is as defined in Figure 13 above.

[0176] In step S1420, the training input x k is applied to the initialized machine learning model NN to generate training outputs.

[0177] Associated target y k The training output M(x k ), or the residual, is quantified by a cost function L. In step S1430, one or more parameters of the model are adjusted in one or more iterations in the inner loop to improve the cost function. For example, the model parameters are adjusted to reduce the residual as measured by the cost function. The parameters include, among other things, the weights of the artificial neural network M.

[0178] The training method then returns to step S1410 in the outer loop, where the next pair of training data is fed. In step S1420, the parameters of the model are adjusted such that the aggregated residual of all pairs considered is reduced, in particular minimized. A cost function quantifies the aggregated residual. Forward-backpropagation or similar gradient-based techniques can be used in the inner loop.

[0179] More generally, the parameters of the model NN are adjusted to improve an objective function F, which may be either a cost function or a utility function. In an embodiment, the cost function is configured to measure an aggregated residual. In an embodiment, the aggregation of the residuals is implemented by summing the residuals of all or a portion of all pairs considered. In particular in an autoencoder type NN network embodiment, when unsupervised learning is used, xk =y k Either there are no pairs where , or there are only "nominal" pairs.

[0180] The method may be implemented on one or more general purpose processing units TS, preferably having processors capable of parallel processing to speed up training. Components of the training system TS may be implemented as one or more software modules and executed on one or more general purpose processing units PU, such as workstations associated with the imagers IA, or on a server computer associated with a group including multiple imagers.

[0181] Although the system USYS and related methods are described primarily with reference to X-radiography, this does not exclude other applications, such as collimation adjustment for radiation therapy, such as multi-leaf collimators or linear accelerator devices, etc. Collimation for external beam radiation therapy or proton therapy (e.g., in ophthalmic therapy, etc.) is also contemplated herein.

[0182] The components of the imaging system USYS may be implemented as one or more software modules and may be executed on one or more general-purpose processing units PU, such as a workstation associated with an imager XI, or on a server computer associated with a group including multiple imagers.

[0183] Alternatively, some or all components of the image processing system IPS may be located in hardware, such as a suitably programmed microcontroller or microprocessor, such as an FPGA (Field Programmable Gate Array), or may be located as a hardwired IC chip, application specific integrated circuit (ASIC), or may be integrated into the imaging system XI. In yet another embodiment, the image processing system IPS may be implemented both partly in software and partly in hardware.

[0184] The various components of the image processing system IPS may be implemented on a single data processing unit PU, or several or more components may be implemented on several different processing units PU, possibly located remotely in a distributed architecture and connected in a suitable communication network, such as a cloud environment or a client-server configuration, web-hosted services, etc. This allows serving several geographically distributed imaging sites in a single medical facility or across several medical facilities.

[0185] One or more features described herein may be configured or implemented as or with circuitry encoded in a computer-readable medium, and / or combinations thereof. Circuitry may include discrete and / or integrated circuits, systems on a chip (SOC), and combinations thereof, machines, computer systems, processors and memories, computer programs.

[0186] In another exemplary embodiment of the invention, a computer program or a computer program element is provided, characterized in that it is arranged for performing, on a suitable system, the method steps of the method according to one of the above-mentioned embodiments.

[0187] The computer program element may therefore be stored in a computing unit which may be part of an embodiment of the present invention. This computing unit may be configured to execute or to direct the execution of the steps of the method. It may further be configured to operate the components of the device. The computing unit may be configured to operate automatically and / or to execute user instructions. The computer program may be loaded into a working memory of a data processor. The data processor may therefore be configured to execute the method of the present invention.

[0188] This exemplary embodiment of the invention covers both computer programs that use the invention from the beginning, and computer programs that, through updates, change existing programs into programs that use the invention.

[0189] Moreover, the computer program element may be capable of providing all the steps required to fulfill the procedures of the exemplary embodiments of the above methods.

[0190] According to a further exemplary embodiment of the present invention, a computer readable medium, such as a CD-ROM, is provided having stored thereon computer program elements as described above.

[0191] The computer program may be stored and / or distributed on a suitable medium (particularly, but not necessarily, a non-transitory medium), such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, or may be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems.

[0192] However, the computer program may also be provided over a network, such as the World Wide Web, and downloaded from such a network into the working memory of a data processor. According to another exemplary embodiment of the invention, a medium for making a computer program element downloadable is provided, the computer program element being configured to perform a method according to one of the above embodiments of the invention.

[0193] It should be noted that the embodiments of the present invention are described with reference to different subject matters. In particular, some embodiments are described with reference to method type claims, while other embodiments are described with reference to device type claims. However, a person skilled in the art will understand from the above and following description that, unless otherwise specified, any combination of features related to different subject matters is considered to be disclosed in this application in addition to combinations of features belonging to one type of subject matter. However, all features are combinable to provide a synergistic effect that is more than a simple addition of the features.

[0194] While the invention has been illustrated and described in detail in the drawings and above, such illustration and description are to be considered illustrative or exemplary and not restrictive. The invention is not limited to the disclosed embodiments. Other variations of the disclosed embodiments can be understood and effected by those skilled in the art to which the claimed invention pertains from the drawings, the disclosure, and the dependent claims.

[0195] In the claims, the term "comprises" does not exclude other elements or steps, and singular elements do not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that several means are recited in mutually different dependent claims does not indicate that a combination of these means cannot be used to advantage. Any reference signs in the claims should not be interpreted as limiting the scope thereof.

Claims

1. 1. A system for facilitating collimator adjustment in x-ray imaging or radiation therapy, the system comprising: an input interface for receiving i) user input data including a first collimator setting for a collimator of an X-ray imaging device; and ii) input data including at least one input image acquired by the X-ray imaging device; a collimator setting estimator that calculates a second collimator setting for the collimator based on the user input data and the at least one input image.

2. The system of claim 1 , wherein the collimator setting estimator is implemented as a trained machine learning model, or the collimator setting estimator calculates the second collimator setting based on output data provided by a trained machine learning model.

3. The system described in claim 2, wherein the machine learning model is an artificial neural network.

4. The system of claim 2 or 3, wherein the output data provided by the machine learning model comprises a feature map or a heat map derived from such a feature map.

5. The system of claim 4 , wherein the collimator setting estimator includes a segmenter that segments the feature map or the heat map into at least one segment, and the calculated second collimator setting is further based on the segment.

6. A system as described in any one of claims 1 to 5, wherein the second collimator setting complements the first collimator setting.

7. The system of claim 1 , wherein the first collimator setting comprises a specification of a geometric point, curve, or line in the input image acquired by the x-ray imaging device.

8. 8. The system of claim 1 , further comprising a user input device for capturing the user input data, the user input device including one or more of a graphical user interface, an eye tracking device, a gesture detection device, and a voice processor.

9. 9. The system of claim 1, wherein the X-ray imaging device is capable of assuming a plurality of different imaging geometries, and the collimator setting estimator adjusts the second collimator setting in response to the X-ray imaging device changing its imaging geometry.

10. The system of claim 1 , wherein the second collimator setting specifies a tightness of collimation.

11. 1. A method for facilitating collimator adjustment in x-ray imaging or radiation therapy, the method comprising: receiving i) user input data including a first collimator setting for a collimator of an x-ray imaging device; and ii) input data including at least one input image acquired by said x-ray imaging device; and calculating a second collimator setting for the collimator based on the user input data and the at least one input image.

12. A computer program product which, when executed by at least one processing device, causes the processing device to perform the method of claim 11.

13. 13. At least one computer readable medium storing the computer program of claim 12 or storing at least a portion of the pre-trained machine learning model.

Images