Multi-organ registration system and method

Abstract

The invention relates to a multi-organ registration system and method, the system comprising: an input module configured to obtain imaging data of at least one tissue portion of at least one organ of a patient at a point in time; a calculation module configured to calculate at least one spatial mapping between the obtained imaging data and corresponding reference imaging data, where the at least one spatial mapping is determined based on the point distance measurements and the feature-based measurements, where the at least one spatial mapping is designed to incorporate adaptive spatial support, and where the at least one spatial mapping is determined based on the point distance measurements and the feature-based measurements. The reference imaging data is obtained from the database or by the input module, corresponding to a different point in time and / or a different patient than the imaging data; a registration module configured to process the obtained imaging data and corresponding reference imaging data using the calculated at least one spatial mapping and generate registration information; and an output module configured to provide the generated registration information.

Description

Technical Field

[0001] The present invention relates to a system for registering imaging data relating to tissue portions of a plurality of organs of a patient. The present invention also relates to a corresponding method, a computer program product and a non-transitory computer-readable data storage medium. Background Art

[0002] Here and in the forthcoming cases, individuals with both male and female identities are included within the term, regardless of the grammatical use of the term.

[0003] Registration of medical imaging data is one of the main tools in digital medicine, allowing the combination of information acquired from different imaging systems or information acquired at different time points or information acquired from different patients. By fusing information acquired from different imaging systems, a comprehensive understanding of the patient's anatomical structure and / or tissue function can be achieved. Alternatively, aligning images acquired at different time points allows the transfer of anatomical and / or functional information between two time points; for example, transferring radiotherapy planning data defined in preoperative computed tomography (CT) to intraoperative cone beam computed tomography (CBCT) immediately before treatment facilitates the precise localization of critical structures of interest as well as target tumors. In addition, registering images acquired from different patients enables atlas reconstruction, population studies (e.g., changes in organs between populations), and disease studies (e.g., changes in anatomical structures due to disease).

[0004] Registration is the process of determining the spatial mapping between a pair of images. For example, one of the images that captures the state / setting of the patient's anatomy of interest is often referred to as the reference image, while the other image is often referred to as the moving image, which is usually of high quality and includes more comprehensive anatomical / functional information.

[0005] Most registration algorithms estimate a single spatial map that aligns the moving image to the reference image, thereby enhancing the reference image with more comprehensive anatomical / functional information from the moving image, although the estimation of a map that aligns the reference image to the moving image is feasible. Furthermore, while the development of algorithms for single-organ registration or full-image registration is quite satisfactory, the registration of multiple organs poses many challenges that have not been addressed to date.

[0006] In multi-organ registration, the degree of deformation, significant motion direction, and deformation speed of a part of an organ or an organ relative to other organs can vary significantly. The varying degree of motion allowed within an organ or the sharp changes in the motion of adjacent organs make modeling such deformations extremely challenging. For example, the myocardium can deform via shear motion rather than via scaling motion, muscles close to the skeleton are stiffer than muscles far from the skeleton, or the upper lobes of the lungs are not as expandable as the lower lobes. In addition, organs that are close to each other can deform almost independently of each other in some cases. For example, the bladder can expand significantly when filled with urine, compared to the almost imperceptible size changes experienced by surrounding structures (such as the uterus, prostate, or rectum).

[0007] Prior art solutions, such as those employing B-spline functions or radial basis functions (RBFs), are suitable for modeling the deformation of tissues at a local level or at a global level. In the first case, the small support of the transformation is not suitable for capturing large coherent motions. In the second case, the large support cannot capture sharp changes in motion. Other existing solutions, such as those based on non-parametric deformations or those based on biomechanical finite element models (FEMs), strongly rely on image quality, deformation boundary conditions and accurate knowledge of the biomechanical properties of the tissue, respectively. Summary of the invention

[0008] Based on this background, the problem to be solved by the present invention is to provide a system for registering multiple organs, which is capable of modeling the complex movements of the organs and their interactions.

[0009] This problem is solved according to the invention by a system having the features of claim 1 and / or by a computer-implemented method having the features of claim 13 and / or by a computer program product having the features of claim 14 and / or by a non-transitory computer-readable data storage medium having the features of claim 15.

[0010] A first aspect of the present invention provides a system for aligning medical imaging data of multiple organs of a patient, the system comprising: an input module, the input module being configured to obtain imaging data of at least one tissue portion of at least one organ of the patient at a time point; a calculation module, the calculation module being configured to calculate at least one spatial mapping between the acquired imaging data (D1) and corresponding reference imaging data, wherein at least one spatial mapping is determined based on point distance measurement and feature-based measurement, wherein at least one spatial mapping is designed to incorporate adaptive spatial support, and wherein the reference imaging data is obtained from a database or by the input module (10) and corresponds to a different time point and / or a different patient than the imaging data; a registration module, the registration module being configured to use the calculated at least one spatial mapping to process the acquired imaging data and the corresponding reference imaging data and generate registration information; and an output module, the output module being configured to provide the generated registration information.

[0011] Imaging data broadly describes any information that can be used for further processing and analysis in medical applications of digital medicine. In the context of the present invention, imaging data may include one or more scanned images of soft tissue and / or bone samples of mammalian organs, such as computed tomography (CT) images or magnetic resonance (MR) images. Some of these images may include selected sub-image portions or crops of larger images, where segmentation or mosaicking has been applied. Imaging data may also include metadata, such as in the form of annotations.

[0012] Image registration is the process by which one or two spatial transformations between two images are estimated, allowing the anatomical / functional information captured by the two images to be combined via, for example, image superposition and / or image fusion. Most registration algorithms seek a single spatial mapping that associates pixels of a moving image (the acquired imaging data, typically with high image quality and more comprehensive anatomical / functional information) with pixels of a reference image (typically of lower quality, but capturing the state / setting of the patient's anatomical structures of interest). Thus, in this case, the spatial mapping maps pixels, sets of pixels, or regions of the moving image domain to corresponding pixels, sets of pixels, or regions of the reference image domain. However, the present invention allows for the estimation of (i) a mapping from a moving image to a reference image, or (ii) a mapping from a reference image to a moving image, or (iii) both a mapping from a moving image to a reference image and a mapping from a reference image to a moving image.

[0013] Point distance measurements and feature-based measurements (or dissimilarity measurements) refer to mathematically expressible comparisons of distances between different points or features of different points. Points here usually refer to pixels or sets of pixels. This measurement can also be called a metric or distance.

[0014] The spatial support of a function is the spatial extension of the function that is non-zero. Adaptive spatial support refers to the possibility to change the spatial support of a function, usually through one or more control parameters.

[0015] If a mapping from the reference image domain to the moving image domain is also estimated, the registration generated by the output module may be in the form of a graphical realization of the aligned or registered moving image domain and / or the aligned or registered reference image domain.

[0016] The different modules and units mentioned in this application are broadly understood as entities capable of acquiring, obtaining, receiving or retrieving general data and / or instructions through a user interface and / or programming code and / or executable program or any combination thereof. In particular, a computing module is suitable for running programming code and executable program and transmitting the results for further processing.

[0017] Therefore, the different modules and units mentioned below or a part thereof may respectively include at least a central processing unit CPU and / or at least one graphics processing unit GPU and / or at least one field programmable gate array FPGA and / or at least one application-specific integrated circuit ASIC and / or any combination of the foregoing. Each of them may also include a working memory operably connected to at least one CPU and / or a non-transitory memory operably connected to at least one CPU and / or the working memory. Each of them may include an application program application (API) or consist of an application program application (API).

[0018] All parts of the system of the present invention can be implemented in hardware and / or software, cable connection and / or wireless and any combination thereof. Any part of the system can include an interface leading to an intranet or the Internet, a cloud computing service, a remote server, etc.

[0019] In particular, each module and unit of the system or the system as a whole can be partially and / or completely implemented in a local device, such as a computer, in a computer system and / or partially and / or completely in a remote system, especially in an edge or cloud computing platform.

[0020] In a system based on cloud computing technology, a large number of devices are connected to the cloud computing system via the Internet. The device can be in a remote facility connected to the cloud computing system. For example, the device can include or consist of equipment, sensors, actuators, robots and / or machines in an industrial setting. The device can be medical devices and equipment in a medical care unit. The device can be a household appliance or office appliance in a residential / commercial establishment.

[0021] Cloud computing systems can enable remote configuration, monitoring, control, and maintenance of connected devices (also commonly referred to as "assets"). In addition, cloud computing systems can facilitate storage of large amounts of data periodically collected from devices, analysis of large amounts of data, and providing insights (e.g., key performance indicators, anomalies) and alerts to operators, field engineers, or owners of devices via a graphical user interface (e.g., a web application). Insights and alerts can enable control and maintenance of devices, resulting in efficient and fail-safe operation of the devices. Cloud computing systems can also enable modification of parameters associated with devices and issuance of control commands via a graphical user interface based on insights and alerts.

[0022] A cloud computing system may include multiple servers or processors (also referred to as "cloud infrastructure") that are geographically distributed and interconnected via a network. A dedicated platform (hereinafter referred to as a "cloud computing platform") is installed on the servers / processors to provide the above functions as a service (hereinafter referred to as a "cloud service"). The cloud computing platform may include multiple software programs executed on one or more servers or processors of the cloud computing system to enable the delivery of requested services to devices and their users.

[0023] One or more application programming interfaces (APIs) are deployed in a cloud computing system to deliver various cloud services to users.

[0024] A second aspect of the present invention provides a computer-implemented method for aligning medical imaging data of multiple organs of a patient, the method comprising the following steps: obtaining imaging data of at least one tissue portion of at least one organ of the patient at a time point; calculating at least one spatial mapping between the acquired imaging data and corresponding reference imaging data, wherein at least one spatial mapping is determined based on point distance measurements and feature-based measurements, wherein at least one spatial mapping is designed to incorporate adaptive spatial support, and wherein the reference imaging data is obtained from a database or by an input module and corresponds to a different time point and / or a different patient than the imaging data; generating alignment information by processing the acquired imaging data and the corresponding reference imaging data using the calculated at least one spatial mapping; and providing the generated alignment information.

[0025] In particular, the method according to the second aspect of the invention may be performed by the system according to the first aspect of the invention. Therefore, the features and advantages disclosed herein in conjunction with the computing device are also disclosed for the method, and vice versa.

[0026] According to a third aspect, the present invention provides a computer program product comprising executable program code configured to, when executed, perform the method according to the second aspect of the present invention.

[0027] According to a fourth aspect, the present invention provides a non-transitory computer-readable data storage medium, the data storage medium comprising executable program code, the executable program code being configured to implement the method according to the second aspect of the present invention when executed.

[0028] The non-transitory computer-readable data storage medium may include or consist of any type of computer memory, in particular semiconductor memory such as solid-state memory. The data storage medium may also include or consist of a hard drive, CD, DVD, Blu-ray disc, USB memory stick, etc.

[0029] According to a fifth aspect, the present invention provides a data stream, the data stream comprising or configured to generate an executable program code, the executable program code being configured to implement the method according to the second aspect of the present invention when executed.

[0030] One of the main ideas of the present invention is to provide a system capable of resolving deformations of multiple organs recorded using medical imaging data. The solution provided by the present invention incorporates a combination of point distance measurements and image feature-based measurements (or dissimilarity measurements) in at least one spatial mapping, the support of which can be adapted to the complexity of the motion and interaction of the organs to be registered. The spatial mapping can map reference imaging data to motion imaging data, or vice versa. These spatial mappings are not exclusive, i.e. the present invention can handle both simultaneously.

[0031] The system as described above allows a simple implementation of a computer-implemented method for registering medical imaging data with multiple organs of a patient. In one step, imaging data containing multiple organs of a patient are obtained at a certain point in time. The imaging data can be generated, for example, by a CT device. In another step, the imaging data is compared with corresponding reference imaging data and at least one spatial mapping is calculated. The reference imaging data can be generated by the same CT device at different points in time or by different imaging systems. The reference imaging data can be obtained from different patients. The spatial mapping is determined from a mathematical function that combines the distance between pixels and the dissimilarity of the image based on general features such as image intensity or modality-invariant image descriptors. In addition, the spatial mapping is designed to be suitable for different supports, so that a sequence of different spatial mappings for different support areas of multiple organs captured in the imaging data can be generated. In a subsequent step, registration information (or registration) of the obtained imaging data is generated based on processing the obtained imaging data using at least one calculated spatial mapping. In another step, the generated registration information is provided.

[0032] One advantage of the present invention is that it provides an adaptable or even self-adaptive tool to capture sharp motion changes and large coherent motions in adjacent organs, which enables registration of medical imaging data of multiple organs to be performed. The system and method of the present invention are modality independent and can be applied to general organs and tissues with different physical properties.

[0033] Another advantage of the present invention is that the present invention incorporates different mechanisms that can reduce computational requirements. One of these mechanisms is a sampling function that selects multiple representative points to infer spatial mapping. Another mechanism is the incorporation of a multi-support function that provides computations utilizing sparse matrices and block matrices, which speeds up computations.

[0034] Advantageous embodiments and developments emerge from the dependent claims and from the description of various preferred exemplary embodiments shown in the drawings.

[0035] According to some embodiments, refinements or variations of embodiments, the point distance measurement comprises a comparison of distances between points of the imaging data and corresponding points of reference imaging data, preferably based on a continuous representation of a point cloud (such as an anisotropic Gaussian mixture), and / or the feature-based measurement comprises a comparison of image features between points of the imaging data and corresponding points of reference imaging data.

[0036] A point cloud is a set of discrete data points that capture the geometry of a three-dimensional structure. Previous work has combined functions with scalable supports using landmark and / or grid-based registrations, where point correspondences are known. According to the present invention, the capabilities of multi-support functions are extended by combining them with a mixture-based registration, in which a set of discrete points is represented by anisotropic Gaussian mixtures. The use of a mixture-based registration to associate different points enables the computation of spatial mappings involving a large number of noisy partially observed surface points with unknown correspondences and unequal numbers of points between groups. This increases the applicability and accuracy with which spatial mappings can be computed.

[0037] According to some embodiments, refinements or variants of embodiments, at least one spatial mapping is determined by minimizing a weighted sum of a point distance measure and a feature-based measure, wherein the weights are predetermined or considered as variables.

[0038] The determination of the spatial mapping can be performed by minimizing a function. In a preferred embodiment of the present invention, the function can be mathematically expressed as:

[0039]

[0040] in, and represents the point cloud in the domain of the moving image and the reference image. The spatial mapping function can be parameterized as:

[0041]

[0042] About the affine coefficients A and Warpage coefficient associated with N Control (or sampling) point and the function U(||p i -x 1，k ||2).

[0043] This function L contains a mapping ψ(X1) from the reference image domain to the moving image domain and an (inverse) mapping ψ′(X0) from the moving image domain to the reference image domain. The calculation module finds the values ​​of the parameters of the functions ψ and ψ′ which minimize the function l.

[0044] The point distance measure is represented by d(·,·), which in some preferred embodiments can be based on the L between the mixed representations of the two point groups X0 and X1 after the transformation. p Distance, as described above. In addition to the fast anisotropic Gaussian mixture distance between point clouds, any other point distance measure that does not assume point correspondence and an equal number of points in each point group may be used, such as the so-called Chamfer and Hausdorff distances.

[0045] The feature-based measure consists of a dissimilarity operator S(·,·) whose argument is the image function The image function may be a scalar function or a multimodal image feature function (vector-valued function). An example of a simple scalar image feature measurement may be image intensity, in which case is the identity operator, Examples of multimodal image feature operators are the modality-independent neighborhood descriptors used in M. Heinrich et al., Medical Image Analysis, 16(7): 1423-1435, 2012 or the self-similar context (SSC) used in M. Heinrich et al., Medical Image Computing and Computer-Assisted Intervention, pp. 187-194, 2013. For example, in using L p The dissimilarity operator S(·,·) can be measured in those cases where the distance or robust Huber distance is sufficient. If the image quality is high and the registration is performed between images of the same imaging modality, the dissimilarity operator S(·,·) can be used, which usually has L p Or the image intensity itself of the Huber loss. Other image features that are often used in image registration can be similarly employed.

[0046] In medical imaging, point clouds are often derived from the segmentation of organs / structures, which may not always be available for all structures in the image, especially for fine structures, such as for the detailed soft tissue contrast of the kidneys captured in magnetic resonance scans or for the detailed identification of vascular structures in contrast-enhanced CT of the liver. Using image features provides a way to compensate for the lack of point cloud information in some cases. However, image intensity has its limitations and does not always represent the anatomical structure faithfully (i.e., it may be confounded by noise, artifacts, low resolution, etc.). For this reason, the weights γ d and γ s Controls the strength of the corresponding point and image dissimilarity terms in the function L.

[0047] According to some embodiments, refinements or variants of embodiments, the computing module includes a machine learning unit, which is configured to implement an unsupervised learning model, preferably a deep learning model or more preferably a deep reinforcement learning model.

[0048] Here and in the coming time, a machine learning unit will be understood as a computerized entity capable of implementing different data analysis methods broadly described under the terms: artificial intelligence, machine learning, deep learning or computer learning. The machine learning unit may include an artificial neural network (or encoder). An artificial neural network is every computing system based on a collection of connected units or nodes clustered into layers capable of sending signals to each other. The artificial neural network may be a multi-layer perceptron network (MLP), a recurrent neural network (RNN), a long short-term memory (LSTM) network, a convolutional neural network (CNN), a denoised diffusion probability model (DDPM) or any other neural network, including those based at least in part on the transformer neural network architecture.

[0049] According to some embodiments, refinements or variants of embodiments, the system of the present invention further comprises a scaling module configured to provide a registration strategy based on a plurality of registration levels, wherein each registration level is characterized by a spatial support.

[0050] The adaptive spatial support incorporated into the spatial mapping can be further utilized by defining a set of registration levels, wherein each registration level uses a different spatial support. Therefore, the scaling module can define a sequence of registration levels based on the characteristics and complexity of the imaging data to be registered, and the registration levels optimize multi-organ registration. The scaling module can be operated manually or automatically. In these embodiments, the user can specify different registration levels to be implemented. In some other preferred embodiments, the registration strategy can be determined by means of an algorithm that can be assisted by a machine learning unit of a computing module, for example. In these embodiments, the sequence of spatial supports can be defined as a hyperparameter to be determined by a deep learning or deep reinforcement learning model. Additionally or alternatively, the support of the registration level can be considered as one of the variables of the spatial mapping to be calculated.

[0051] An example of a registration strategy may be as follows: a first registration level with a large support is used as a coarse registration level, followed by registration levels with smaller supports for finer registration. A sequence of registration levels with decreasing support allows modeling of different degrees of deformation experienced by different organs (e.g., smooth and nearly global motion of the spine and local motion of the bladder due to urine filling), as well as modeling of sudden changes in motion of adjacent structures (e.g., motion of a lung lobe relative to the motion of a rib).

[0052] According to some embodiments, refinements or variants of embodiments, the system of the present invention further comprises a sampling module configured to select a set of sampling points for evaluating point distance measurement and / or feature-based measurement.

[0053] In order to optimize the computational requirements of the computational module, dissimilarity image functions and / or point distance measurements may be evaluated on a set of salient points. Point salient may be determined using shape (e.g., curvature) and / or texture information derived from image intensity and / or image segmentation (distance transform) and / or its derivatives. For example, a sampling function may be defined based on the first and second order derivatives of the distance transform. Sampling points may be defined as sampling points whose first order derivative is zero, or as sampling points whose second order derivative has all positive eigenvalues, all negative eigenvalues, or non-zero (both positive and negative) eigenvalues ​​for local minima, local maxima, or saddle points, respectively.

[0054] According to some embodiments, refinements or variants of embodiments, the system of the invention further comprises a refinement module configured to select at least one sub-region of the acquired imaging data for one or more spatial maps to be calculated.

[0055] To further improve the alignment in challenging areas, in some preferred embodiments, the present invention provides sub-region registration refinement. The sub-regions and reference image domains in The sub-regions in are defined as bounding boxes or unions of organ segmentation. The refinement involves determining the parameters of ψ and / or ψ′ that minimize the function L over the sample points from the corresponding sub-region interior.

[0056] According to some embodiments, refinements or variants of embodiments, the refinement module is configured to define the sub-regions by requiring no deformation at the boundaries of the sub-regions.

[0057] The spatial mapping within a sub-region for sub-region registration refinement can be defined as and The hard constraint on the boundary is thus ψ(x) = ψ(x) / ... 1，j )=x 1，j and ψ′(x 0，j )=x 0,j

[0058] In some embodiments of the present invention, soft constraints may be imposed instead of hard constraints to allow more flexibility at the boundaries. The soft constraints may be defined as:

[0059]

[0060] in, yes The union of yes The union of .

[0061] According to some embodiments, refinements or variations of the embodiments, the spatial support is controlled by a scaling parameter, which is set by the user as a hyperparameter or considered as a variable. The scaling parameter is a scalar value introduced in the spatial mapping, and it generates a family of spatial mapping functions based on the value of the spatial support. As part of the minimization of the above function L, we can fix the value of the spatial support or infer the value at test time.

[0062] According to some embodiments, refinements or variants of the embodiments, the spatial mapping is selected from a set of multi-supported compactly supported radial basis functions (CSRBF). RBF is particularly effective when it is interpolated at points that are not uniformly dispersed. In the literature, we often find the Wendland, Wu, Bahmann or Gneiting forms, which are respectively given by:

[0063]

[0064] where the parameter α is an example of a scaling parameter that controls the adaptive spatial support. Thus, the spatial mapping takes the form

[0065]

[0066] Among them, U(||p i -x 1，k ||2;α) can be given by any CSRBF, including the examples cited above.

[0067] According to some embodiments, refinements or variants of embodiments, the spatial mapping is determined by using geometric priors and / or transformation constraints and / or organizational properties comprising at least a topology preserving constraint.

[0068] In addition to the features already mentioned, the determination of the spatial mapping can also be improved in terms of geometric accuracy if some priors are provided. These priors can include landmark adaptation constraints Constraints via sliding interface Tissue-related motion, incompressibility constraints Topology Preserving Constraints (such as reversibility constraints) and rigid body motion constraints (which may take into account, inter alia, orthogonality, appropriateness and / or affinity constraints ). These geometric priors can be weighted and added separately as:

[0069]

[0070] The function to be minimized can then be modified to take these priors into account as:

[0071]

[0072] If soft bounds are imposed, they can be obtained by Add to the prior function R to achieve.

[0073] According to some embodiments, refinement or embodiment variants, system of the present invention is configured to be implemented in a registration system supported by a global system, wherein system of the present invention is suitable for providing adaptive spatial support to a registration system supported by a global system. Features of the present invention can be added to the existing methods of deploying B-spline functions or RBF functions (which themselves do not provide adaptive spatial support), thereby improving the ability of the multi-organ motions that solve challenges. In these embodiments, the computer program product according to the third aspect of the present invention can be provided as an add-on component of the registration systems supported by these global systems.

[0074] Although here, above and below, some functions are described as being performed by modules or units, it should be understood that this does not necessarily mean that such modules or units are provided as separate entities. In the case where one or more modules or units are provided as software, the modules or units may be implemented by program code parts or program code fragments, which may be different from each other, but may also be intertwined or integrated with each other.

[0075] Similarly, where one or more modules or units are provided as hardware, the functionality of one or more modules or units may be provided by the same hardware component, or the functionality of several modules or units may be distributed across several hardware components, which do not necessarily correspond to the modules or units. Therefore, any device, system, method, etc. that exhibits all features and functions attributed to a particular module or unit should be understood to include or implement said module or unit. It is particularly possible that all modules or units are implemented by program code executed by a computing device, such as a server or a cloud computing platform.

[0076] The above-described configurations and developments may be combined where appropriate, and the embodiments may be combined with one another as necessary as long as this is reasonable.

[0077] Other possible configurations, developments and embodiments of the present invention also include combinations of features of the present invention that are not explicitly mentioned, which have been described above or are described below with reference to the embodiments. In particular, in this case, those skilled in the art will also add various aspects as improvements or supplements to the basic form of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] The invention is described in more detail below based on the embodiments shown in the schematic diagrams of the accompanying drawings, in which:

[0079] Figure 1 is a schematic diagram of a multi-organ registration system for registering medical imaging data of multiple organs of a patient according to an embodiment of the present invention;

[0080] Figure 2is a block diagram illustrating an exemplary embodiment of a computer-implemented method for registering medical imaging data with multiple organs of a patient according to an embodiment of the present invention;

[0081] Figure 3 is a schematic example of point sampling for computing image feature dissimilarity measures, where the structure of interest cannot be delineated;

[0082] Figure 4 is an illustrative example of a registration strategy for a patient's chest using adaptive spatial support according to some embodiments of the present invention;

[0083] Figure 5 is a description of the multi-organ motion estimated by the computational module of the present invention, demonstrating the ability of the spatial mapping to capture the different degrees of deformation experienced by neighboring organs in the thorax and abdomen and to model the highly localized deformations and nearly independent large translations of different organs in the body cavity;

[0084] Figure 6 is a schematic comparison of tissue-related deformations obtained using a conventional registration system and an embodiment of a system according to the present invention;

[0085] Figure 7 is a schematic depiction of a globally supported registration system and a system according to the first aspect of the invention, wherein the system is implemented in the globally supported registration system and adapted to provide adaptive spatial support thereto;

[0086] Figure 8 is a schematic block diagram illustrating a computer program product according to an embodiment of the third aspect of the present invention; and

[0087] Fig. 9 is a schematic block diagram illustrating a non-transitory computer-readable data storage medium according to an embodiment of the fourth aspect of the present invention. DETAILED DESCRIPTION

[0088] The accompanying drawings are intended to provide a further understanding of the embodiments of the present invention. The accompanying drawings illustrate the embodiments and together with the description help explain the principles and concepts of the present invention. Other embodiments and many of the advantages mentioned become apparent with reference to the accompanying drawings. The elements in the drawings are not necessarily shown to scale.

[0089] In the drawings, similar, functionally equivalent and identical operating elements, features and components are provided with similar reference numerals in each case, unless stated otherwise.

[0090] The numbering of the steps in the method is for the convenience of its description. These numbers do not necessarily imply a certain order of the steps. In particular, multiple steps can be performed simultaneously.

[0091] The detailed description includes specific details for the purpose of providing a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without these specific details.

[0092] Figure 1 A schematic diagram of a multi-organ registration system 100 for registering medical imaging data of a plurality of organs of a patient according to an embodiment of the present invention is shown.

[0093] Figure 1 The system 100 shown in FIG. 1 includes an input module 10 , a calculation module 20 , a registration module 30 , an output module 40 , a scaling module 50 , a sampling module 60 , and a refinement module 70 .

[0094] The input module 10 is configured to obtain motion imaging data D1 and reference data of at least one tissue portion of at least one organ of a patient generated at different time points or using different imaging systems. In some embodiments of the present invention, the reference data may be of different patients. The imaging data D1 and the reference imaging data may be derived from, for example, a magnetic resonance scan or a computed tomography scan of a patient's chest.

[0095] The calculation module 20 is configured to calculate one or more spatial mappings between the acquired imaging data D1 and corresponding reference imaging data.

[0096] The spatial mapping can be determined by minimizing the following function

[0097]

[0098] According to the invention, it includes point distance measurement and feature-based measurement. Transformation priors can also be incorporated. Point distance measurement includes a comparison of the distance between a point of the imaging data D1 and a corresponding point of the reference imaging data, preferably based on a comparison of a point cloud representation. Feature-based measurement in turn includes a comparison of image features between a point of the imaging data D1 and a corresponding point of the reference imaging data.

[0099] Spatial mapping also incorporates adaptive spatial support by which the spatial extension of applicable functions can be varied.

[0100] The calculations of the calculation module 20 may be based on one or more algorithms. To calculate the point distance measure, a plurality of control (or sampling) points may be selected. Likewise, the calculation of the feature-based measure may be accomplished using a plurality of sampling points selected according to some criteria. In some preferred embodiments, the function to be minimized comprises a weighted sum of the point distance measure and the feature-based measure, wherein the weighting factors may be determined as part of the minimization process.

[0101] The spatial support of the spatial mapping can be controlled using multiple parameters. In some preferred embodiments of the present invention, each spatial mapping is parameterized using a compact support radial basis function (CSRBF), adapted in such a way that the spatial support can be measured using a single scalar parameter. This parameter can be predetermined or fixed by any algorithm used implemented in the computation module 20.

[0102] In some embodiments of the present invention, such as Figure 1 In the embodiment depicted in , the computing module 20 includes a machine learning unit 210, which is configured to implement an unsupervised learning model, preferably a deep learning model, or more preferably a deep reinforcement learning model.

[0103] The registration module 30 is configured to process the imaging data D1 obtained by the input module 10 and the reference imaging data using the spatial mapping(s) calculated by the calculation module 20 , and thereby generate registration information R1 .

[0104] The output module 40 is configured to provide the generated registration information R1. The output module 40 may include a user interface, wherein the generated registration information R1 may be visualized by a user.

[0105] exist Figure 1 In the embodiment shown in , the system 100 of the present invention further includes a scaling module 50 , a sampling module 60 and a refinement module 70 .

[0106] The scaling module 50 is configured to provide a registration strategy based on multiple registration levels, wherein each registration level is characterized by a spatial support. The sampling module 50 can therefore design a sequence of spatial supports for optimal registration. The sequence can be transmitted to the computation module 20 so that one or more spatial maps are generated for each registration level.

[0107] The sampling module 60 is configured to select a set of sampling points for evaluating point distance measures and / or feature-based measures. By this selection of points, the computational requirements of the computational module 20 can be optimized. The selection of salient points can be performed according to different criteria. One such possible criterion can be to define a sampling function based on the first and / or second order derivatives of the distance transform. The sampling points can be defined as those sampling points whose first order derivative is zero or whose second order derivative has all positive eigenvalues, all negative eigenvalues, or non-zero eigenvalues ​​(with positive and negative values) that represent local minima, local maxima, or saddle points of the structure of interest, respectively. This selection of sampling points can be transmitted to the computational module 20 in order to compute point distance measures and / or feature-based measures of the spatial map.

[0108] The refinement module 70 is configured to select at least one sub-region of the acquired imaging data D1 for the spatial mapping to be calculated. With the refinement module 70, the alignment can be improved in particular in challenging regions of the imaging data, where the motion of different organs must be carefully characterized (e.g., motions presenting sudden changes in amplitude or direction, such as the motion of a lung lobe relative to a rib). The sub-regions may be defined by the requirement that there is no deformation at their boundaries. The selection of the sub-regions may be transmitted to the calculation module 20, so that one or more spatial mappings are generated for each sub-region, i.e. the parameters of the functions ψ and ψ′ are determined, which minimize the function L on the sampling points from the interior of the corresponding sub-region.

[0109] Figure 2 is a block diagram illustrating an exemplary embodiment of a computer-implemented method for registering medical imaging data of multiple organs of a patient according to an embodiment of the present invention. The method may preferably utilize reference Figure 1 The multi-organ registration system 100 is implemented, and the method includes multiple steps.

[0110] In a step S1, imaging data D1 of at least one tissue portion of at least one organ of a patient at a time point is obtained. The imaging data D1 can be obtained by a medical scanning imaging device, such as a computed tomography (CT) device or a magnetic resonance imaging (MRI) device. Reference imaging data can also be obtained, which can be obtained at a different time point or obtained using a different imaging system or obtained from a different patient.

[0111] In another step S2, one or two spatial mappings (i.e., ψ or ψ′, or both ψ and ψ′) between the motion imaging data and the corresponding reference imaging data D1 are calculated. These spatial mappings can be determined based on the minimization of a function comprising point distance measurements, feature-based measurements under transformation constraints, the point distance measurements and feature-based measurements being input to the function as weighted sums in some preferred embodiments. The spatial mappings ψ and ψ′ are additionally adapted to be able to cover different supports, so that step S2 can be repeated for the same imaging data taking into account different adjustments of the spatial supports of the spatial mappings. In some embodiments of the invention, the sequence of such alignment levels is determined by a scaling module 50. The alignment strategy can be implemented or supported by a machine learning approach.

[0112] In step S3, registration information R1 of the acquired motion imaging data D1 and the reference imaging data is generated. The registration information R1 is based on processing the motion imaging data D1 and the reference imaging data using the calculated one or more spatial mappings.

[0113] In step S4, the generated registration information R1 is provided. The registration information R1 may be suitable for visualization by a user, for example via a user interface.

[0114] Figure 3 A schematic example of sampling points P1 to P3 in a region where point cloud registration is impaired and information about, for example, image intensity can be used for registration is shown. A plurality of such exemplary sampling points P1 to P3 for computing feature-based measurements of a spatial map is shown. Figure 3 The right picture is Figure 3 Magnified view of the square area in the left image.

[0115] Figure 4 A schematic example of a registration strategy for a patient's chest using adaptive spatial support according to some embodiments of the present invention is shown.

[0116] exist Figure 4 The image in the upper left part shows the superposition of motion imaging data D1 and reference imaging data of the patient's chest obtained by using a CT device, wherein the motion image shows the lobes at the time point of full inhalation and the reference image shows the lobes at the time point of full exhalation. The image shows the patient's right upper lobe O1, right middle lobe O2, right lower lobe O3, left upper lobe O4, left lower lobe O5, heart O6 and spleen O7. The reference imaging data and the motion imaging data D1 are obtained from the same patient at different times. In the motion imaging data D1, the lobes O1-O5 are much smaller than those in the reference imaging data. In contrast, the size of the heart O6 and the spleen O7 has hardly changed, but due to the patient's respiratory movement, large, almost independent translations of the heart O6 and the spleen O7 can be observed according to the difference in the spatial positions of the heart and the spleen in the motion imaging data D1 and the reference imaging data.

[0117] Figure 4 The upper right, lower left and lower right images of show a possible registration strategy according to the principles of the present invention to provide registration of imaging data D1 and reference imaging data taking into account the presence of multiple organs O1 to O7. The registration strategy can be generated using the zoom module 50 of the present invention.

[0118] Figure 4 The upper right image of shows a first registration level RL1 with a large support. This may correspond, for example, to a scaling parameter α, which controls the spatial support of the spatial mapping, taking the value α=1 (for the image space domain whose coordinates in all x, y and z directions are normalized to the range [-1, 1]). This first registration level RL1 corresponds to a coarse registration level, aiming mainly at capturing the almost global coherent motion of the organs O1 to O7. The corresponding spatial mapping is calculated by the calculation module 20 of the invention.

[0119] Figure 4The lower left image corresponds to a finer second registration level RL2, where α < 1. In this level, due to the smaller support of the spatial map, the registration is focused on aligning individual structures of interest, as can be noticed from the improved alignment of each structure of interest O1 to O7. Furthermore, due to the smaller support (and sliding interface constraints), the spatial map can better capture the different local expansions experienced by each lung lobe and the nearly independent translations experienced by the heart O6 and spleen O7.

[0120] Figure 4 The lower right image corresponds to the third registration level RL3, where α <<1. In this level, local misalignment and motion of detailed structures (including structures not represented by point clouds but included in the computation of image feature dissimilarity measures) are corrected due to the refined support of the spatial mapping. In addition, the use of this registration level RL3 guides the transformation to better capture the independent motion of organs (e.g., the motion of lobes O1-O5 relative to the heart O6 and spleen O7) and sudden changes in the amplitude and direction of motion (e.g., large motion of the lower lobe relative to the upper lobe). With this third registration level RL3, the relative motion of all organs O1 to O7 can be specifically resolved.

[0121] For each registration level RLi (RL1 to RL3), the calculation module 20 calculates the corresponding spatial mapping ψ i and / or ψ′ i , the spatial mapping minimizes the function L. The spatial mapping ψ i and ψ′ i The parameter values ​​of will be different for different registration levels RL1 to RL3, since they correspond to different values ​​of the scaling parameter α. i and ψ′ i The CSRBF functions used in the parameterization of do not have to be the same. The final deformation is the composite of the spatial maps estimated in each registration level, e.g. (The same applies to ψ′).

[0122] Based on the spatial maps calculated for the different registration levels RL1 to RL3 , the output module 30 is able to generate registration information R1 of the initial imaging data D1 .

[0123] Figure 5 Depicted is an example of a displacement field (deformations experienced from a motion image domain compared to a reference image domain) capturing complex motion of multiple organs in the chest and abdomen obtained using the system 100 according to an embodiment of the present invention.

[0124] Figure 5The left figure depicts the motion parameterized by a multi-support spatial mapping of organs in the thoracic cavity between full inspiration and full expiration. The multi-support spatial mapping demonstrates the ability to model the different degrees of deformation experienced by the different lobes O1-O5, wherein the lower lobes O3 and O5 exhibit better expansion and compression capabilities compared to the upper lobes O1 and O4 and the middle lobe O2. In addition, the spatial mapping can model the effects of the motion of the lower lobes O3 and O5 on the almost independent large translations of the liver and spleen, wherein the amplitude of the local motion is >75mm. Therefore, the multi-support spatial mapping obtained using an embodiment of the system 100 according to the present invention can model the complex independent motion of the organs in the thoracic cavity.

[0125] Figure 5 The right figure depicts the motion parameterized by the multi-support spatial mapping of organs within the abdominal cavity during dual-energy CT imaging. Since the difference in the acquisition time of the low-energy CT and high-energy CT images is very small, the only motion observed is that of the upper abdominal organs (e.g., liver, spleen, and kidneys) due to respiratory motion, as can be noticed from the misalignment of the contours of the organs (e.g., liver O8 and spleen O7) in the motion imaging data and the reference imaging data. Due to the multiple supports, the spatial mapping demonstrates its ability to model the highly localized motion of the liver O8 and spleen O7 in a stationary environment, as can be observed from the displacements with zero amplitude everywhere except for the liver and spleen.

[0126] Figure 6 A schematic comparison of deformations (displacement fields) obtained with a conventional registration system and with an embodiment of the system according to the invention is shown.

[0127] Figure 6 The leftmost image of shows the deformation calculated using a conventional registration system without tissue association and geometric constraints. The absence of such constraints results in the inability to capture the stiffness of the vertebrae, as shown in the second image from the left, which corresponds to a magnified view of the spine (central area) of the leftmost image.

[0128] from Figure 6 The third image from the left of shows the deformation obtained with an embodiment of the system 100 according to the present invention. Compared to registration with a conventional registration system, the stiffness of the bone structure is successfully captured. Figure 6 The rightmost image of shows a magnified view of a vertebra, the stiffness of which has been preserved. This can be achieved using the principles described for the present invention, in particular using a combination of point distances, feature-based measurements and prior knowledge about the tissue and deformation properties in a multi-support space mapping, which enables the system to have adaptive spatial supports and provide different registration levels, as well as the possibility to impose geometric priors and define sub-regions for registration refinement.

[0129] Figure 7 A globally supported registration system 500 and a system 100 according to a first aspect of the invention are shown. The globally supported registration system 500 can be any registration system whose spatial mapping is based on medical imaging data with a fixed supported B-spline function or RBF function. The system 100 is suitable for providing adaptive spatial support to the globally supported registration system 500, thereby improving its ability to resolve challenging multi-organ motions. The system 100 can be wirelessly coupled to the globally supported registration system 500 or connected to the globally supported registration system 500 using a cable.

[0130] Figure 8 A schematic block diagram illustrating a computer program product 300 according to an embodiment of the third aspect of the invention is shown. The computer program product 300 comprises an executable program code 350 which, when executed, is configured to implement a method according to any embodiment of the second aspect of the invention, in particular as already described with respect to the previous figures.

[0131] Fig. 9 A schematic block diagram of a non-transitory computer-readable data storage medium 400 according to an embodiment of the fourth aspect of the invention is shown. The data storage medium 400 comprises an executable program code 450 which, when executed, is configured to implement a method according to any embodiment of the second aspect of the invention, in particular as already described with respect to the previous figures.

[0132] The non-transitory computer-readable data storage medium may include or consist of any type of computer memory, in particular semiconductor memory such as solid-state memory. The data storage medium may also include or consist of a CD, DVD, Blu-ray disc, USB memory stick, etc.

[0133] The above description of the disclosed embodiments is only an example of possible implementations, which are provided to enable any person skilled in the art to implement or use the present invention. Various changes and modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present invention is not limited to the embodiments shown herein, but should be given the widest scope consistent with the principles and novel features disclosed herein. Therefore, the present invention is not subject to any restrictions except according to the appended claims.

Claims

1. A system (100) for registering medical imaging data of a plurality of organs (O1 to O8) of a patient, the system (100) comprising: An input module (10) configured to obtain imaging data (D1) of at least one tissue portion of at least one organ of the patient at a time point; a calculation module (20) configured to calculate at least one spatial mapping between the acquired imaging data (D1) and corresponding reference imaging data, wherein the at least one spatial mapping is determined based on point distance measurements and feature-based measurements, wherein the at least one spatial mapping is designed to incorporate adaptive spatial support, and wherein the reference imaging data is obtained from a database or obtained by the input module (10) and corresponds to a different time point and / or a different patient than the imaging data (D1); a registration module (30) configured to process the acquired imaging data (D1) and the corresponding reference imaging data using the calculated at least one spatial map and generate registration information (R1); and An output module (40) is configured to provide the generated registration information (R1).

2. The system according to claim 1, wherein: The point distance measurement comprises a comparison of distances between points of the imaging data (D1) and corresponding points of the reference imaging data, preferably based on a continuous representation of a point cloud, and / or the feature-based measurement comprises a comparison of image features between points of the imaging data (D1) and corresponding points of the reference imaging data.

3. A system according to any one of the preceding claims, wherein: The spatial mapping is determined by minimizing a weighted sum of the point distance measure and the feature-based measure, wherein weights are predetermined or considered as variables.

4. A system according to any one of the preceding claims, wherein: The computing module (20) includes a machine learning unit (210), which is configured to implement an unsupervised learning model, preferably a deep learning model, or more preferably a deep reinforcement learning model.

5. The system according to any of the preceding claims, further comprising a scaling module (50) configured to provide a registration strategy based on a plurality of registration levels (RL1, RL2, RL3), wherein Each registration level (RL1, RL2, RL3) is characterized by a spatial support.

6. The system according to any of the preceding claims, further comprising a sampling module (60) configured to select a set of sampling points (P1, P2, P3) for evaluating the point distance measurement and / or the feature-based measurement.

7. The system according to any of the preceding claims, further comprising a refinement module (70) configured to select at least one sub-region of the acquired imaging data (D1) for the spatial map to be calculated.

8. The system according to claim 7, wherein: The refinement module (70) is configured to define the sub-regions by requiring that there be no deformation at the boundaries of the sub-regions.

9. A system according to any one of the preceding claims, wherein: The spatial support is controlled by a scaling parameter which is set by the user as a hyperparameter or considered a variable.

10. A system according to any one of the preceding claims, wherein: The spatial mapping is selected from a set of multi-supported compactly supported radial basis functions.

11. A system according to any one of the preceding claims, wherein: The spatial mapping is determined by using geometric priors and / or transformation constraints and / or tissue properties including at least topology preserving constraints.

12. The system according to any of the preceding claims, configured to be implemented into a globally supported registration system (500), wherein: The system (100) is adapted to provide adaptive spatial support to the globally supported registration system (500).

13. A computer-implemented method for registering medical imaging data of a plurality of organs (01 to 08) of a patient, the method preferably being implemented using a system (100) according to any one of claims 1 to 12, the method comprising the following steps: Obtaining (S1) imaging data (D1) of at least one tissue portion of at least one organ of the patient at a time point; calculating (S2) at least one spatial mapping between the obtained imaging data (D1) and corresponding reference imaging data, wherein the at least one spatial mapping is determined based on point distance measurements and feature-based measurements, wherein the at least one spatial mapping is designed to incorporate adaptive spatial support, and wherein the reference imaging data is obtained from a database or by the input module (10) and corresponds to a different time point and / or a different patient than the imaging data (D1); generating (S3) registration information (R1) by processing the obtained imaging data (D1) and the corresponding reference imaging data using the calculated at least one spatial map; and The generated registration information (R1) is provided (S4).

14. A computer program product (300) comprising an executable program code (350) configured to perform the computer-implemented method according to claim 13 when executed.

15. A non-transitory computer-readable data storage medium (400) comprising executable program code (450) configured to perform the computer-implemented method according to claim 13 when executed.

Images