Medical imaging system and method of operation thereof, method of locally deforming a three-dimensional model of a simulated organ

Abstract

The invention relates to a medical imaging system and a method of operating the same, a method of simulating local deformation of a three-dimensional model of an organ. According to these methods, point clouds are computed from intraoperative image data captured before and after a local deformation motion (18) of an organ (10). Geometric feature vectors (40) are assigned to immobile points (20, 21, 22) whose positions do not change due to the local deformation motion (18). Difference vectors (42) indicating changes of the geometric feature vectors (40) due to changes of positions of mobile points (30, 31, 32) in the vicinity of the immobile points (20, 21, 22) are used to estimate the local deformation motion (18) of the organ (10).

Description

Technical Field

[0001] This invention relates to methods for operating medical imaging systems to estimate local deformations of organs. The invention also relates to methods for simulating local deformations of three-dimensional models of organs, medical imaging systems, and software program products. Background Technology

[0002] Laparoscopic surgery (also known as laparoscopic procedures or laparoscopic examinations) offers numerous advantages over traditional open surgery, such as smaller incisions and shorter recovery times. Intraoperative navigation plays a crucial role in laparoscopic surgery. However, this navigation presents several challenges. Organs cannot be palpated, preventing surgeons from obtaining tactile feedback. Consequently, the location of tumors cannot be felt. Furthermore, laparoscopic ultrasound is the only available technique to visualize subsurface structures beneath the surface of the target organ, such as veins, arteries, and tumors. Laparoscopic ultrasound typically provides only 2D images, which need to be integrated into the surgeon's mind to reference the actual location of the 2D ultrasound slices in 3D space.

[0003] To address this issue, preoperative image data of the organ, such as computed tomography (CT) or magnetic resonance (MR) images, is typically captured before surgery. This preoperative image data is transformed into a three-dimensional model of the organ, and then this three-dimensional model is aligned with intraoperative laparoscopic image data. This allows additional information from the preoperative images to be added to the intraoperative images. The alignment of the model with the intraoperative images is called image registration.

[0004] However, in preoperative image data, organs are represented in their natural, undeformed state. In contrast, during laparoscopic surgery, the target organ typically undergoes deformation due to the pressure caused by injected gas. Furthermore, due to the limited space in the abdominal cavity, mobilization techniques are required to move portions of the target organ to the appropriate location for resection. These mobilization techniques also involve local deformation of the organ. Therefore, the local deformation of the organ needs to be applied to the model to obtain accurate registration results before registration.

[0005] Biomechanical modeling based on the finite element method is commonly used to simulate deformation. However, these biomechanical models require prior knowledge of several parameters, including organ stiffness. If these parameters are unknown or deviate from standard values, for example due to organ disease, this method cannot accurately predict the deformation.

[0006] Another approach utilizes intraoperative imaging techniques such as CT fluoroscopy to model the deformation. However, to achieve a correspondence between intraoperative and preoperative images, reference markers must be placed on the organ prior to preoperative imaging, requiring additional surgical steps. Furthermore, cone-beam CT is necessary during laparoscopic surgery. Moreover, registration of fluoroscopic images with the intraoperative surface requires 2D-to-3D registration methods, which are technically more difficult than 3D-to-3D registration. Summary of the Invention

[0007] The purpose of this invention is to provide a simple, rapid, reliable and inexpensive method for estimating local deformation of organs, especially during laparoscopic surgery.

[0008] This objective is achieved by a method for operating a medical imaging system to estimate local deformations of an organ, wherein the medical imaging system includes a processing unit, wherein the processing unit...

[0009] - Receive first intraoperative image data of the organ captured at a first time point and second intraoperative image data of the organ captured at a second time point, wherein the first time point is before the local deformation movement of the organ and the second time point is after the local deformation movement of the organ.

[0010] - A first intraoperative point cloud is calculated based on the first intraoperative image data, and a second intraoperative point cloud is calculated based on the second intraoperative image data, wherein the first intraoperative point cloud includes information regarding the spatial location of image points representing the organ surface at the first time point, and wherein the second intraoperative point cloud includes information regarding the spatial location of the image points representing the organ surface at the second time point.

[0011] - Classify at least one image point in the first intraoperative point cloud and its corresponding point in the second intraoperative point cloud as fixed points, wherein the spatial location of the fixed points is the same at the first time point and at the second time point.

[0012] - Calculate geometric feature vectors and assign them to at least one fixed point in the first intraoperative point cloud and the second intraoperative point cloud, wherein the geometric feature vector of the fixed point indicates at least one geometric relationship between the fixed point and at least one of its neighboring image points.

[0013] - Calculate the difference vector at at least one fixed point, where the difference vector is the difference between the geometric eigenvector at the first time point and the geometric eigenvector at the second time point.

[0014] - At least one image point in the first intraoperative image data is designated as a neighboring moving point, wherein the neighboring moving point is located in a predefined space centered on the at least one stationary point, and wherein the spatial position of the neighboring moving point changes between the first time point and the second time point.

[0015] - Estimate the positional change of the at least one neighboring moving point based on the difference vector of at least one fixed point, and use the positional change to estimate the local deformation of the organ.

[0016] Advantageously, this method allows for the estimation of local organ deformation without requiring additional imaging techniques or knowledge of organ-specific parameters, such as their mechanical stiffness. First intraoperative image data is captured at a first time point before the local deformation, and second intraoperative image data is captured at a second time point after the local deformation. Both the first and second intraoperative image data are transformed into point clouds. This allows the system to track which points changed their spatial position (i.e., moving points) and which points did not change their position (i.e., stationary points).

[0017] In particular, the processing unit classifies image points in the first and second intraoperative point clouds as fixed points by comparing the spatial positions of image points in the two point clouds and classifying those image points with the same spatial position as fixed points. Conversely, image points not classified as fixed points are classified as moving points.

[0018] In this specification, if an image point in the second intraoperative point cloud represents the same part of the organ surface as an image point in the first intraoperative point cloud, then the image point in the second intraoperative image data is the corresponding point to the image point in the first intraoperative point cloud.

[0019] After classifying one or more fixed points, the system assigns geometric feature vectors to one or more fixed points in two point clouds. The geometric feature vectors represent the geometric relationship between an image point and its neighbors. This geometric relationship can be, for example, the orientation of its surface normal, the angle between its surface normal and the normals of neighboring surfaces, or the distance to a neighboring point in 3D space.

[0020] In particular, the geometric feature vector comprises multiple vector components corresponding to the number of geometric relations it indicates. The value of each vector component specifically indicates the number of neighboring points to which the corresponding geometric relation applies.

[0021] The neighboring points of the first image point are image points located in a predefined space centered on the first image point. By changing the size of the predefined space, the sensitivity of the geometric feature vector of the first image point to its surrounding environment can be affected. In particular, the predefined space is a circle or sphere with a predefined radius centered at a fixed point. The value of the predefined radius can be set by the user as needed.

[0022] This method relies on the fact that the geometric eigenvectors of points in a point cloud depend on their neighboring points. If the position of at least one of the neighboring points of a fixed point changes (i.e., it becomes a moving point), then even the geometric eigenvector of that fixed point will change. Therefore, by determining the change in the geometric eigenvector (i.e., the difference vector), the change in the spatial position of the neighboring moving points can be estimated. In this way, the local deformation motion of an organ can be estimated by examining only the fixed point and its difference vector. It is not necessary to track the motion of the moving point itself.

[0023] Because the migration steps are highly standardized, the identification of local deformation motions becomes easier. This means that each typical local deformation motion will cause a change in the characteristics of the geometric feature vectors of the fixed points. By identifying these changes in the characteristics of the geometric feature vectors, the system can easily infer the local deformation motion. In particular, the local deformation motion is estimated by integrating the difference vectors of multiple fixed points near the deformed part of the organ.

[0024] According to the implementation method, in order to estimate the positional change of at least one neighboring moving point, the processing unit simulates the spatial positional change of the at least one neighboring moving point until the change of the obtained geometric feature vector matches the difference vector.

[0025] In particular, multiple fixed points are classified, assigned geometric feature vectors, and used to estimate the positional changes of at least one of their neighboring moving points (especially all of their neighboring moving points). According to an implementation, the processing unit calculates the geometric feature vectors and assigns them to each point in a first intraoperative point cloud and / or a second intraoperative point cloud.

[0026] In particular, the difference vector is calculated by subtracting the geometric eigenvector at the second time point from the geometric eigenvector at the first time point, and vice versa. Specifically, the difference vectors of multiple fixed points near the moving portion of the organ are calculated and used to estimate the positional changes of their neighboring moving points. Changes in the geometric eigenvectors will be most significant at fixed points near the locally deformed portions of the organ. Therefore, it is preferable to select fixed points near the locally deformed portions of the organ to estimate the local deformation motion.

[0027] Geometric eigenvectors and / or difference vectors can be represented as histograms. The calculation and evaluation of the difference vectors can therefore be performed using histogram analysis.

[0028] Preferably, in order to calculate the first intraoperative point cloud and / or the second intraoperative point cloud, optical and / or electronic markers are placed on the surface of the organ, and the position of the markers is determined using a tracking system, wherein the processing unit uses the position of the markers to calculate the position of image points in the first intraoperative point cloud and / or the position of image points in the second intraoperative point cloud.

[0029] In this way, the processing unit can access the depth information of image points in the first and second intraoperative image data. This depth information is used to calculate the point cloud.

[0030] According to an embodiment, the medical image processing system includes an intraoperative image acquisition device that records intraoperative image data, particularly the first intraoperative image data and / or the second intraoperative image data, and transmits the intraoperative image data to the processing unit. Specifically, the intraoperative image acquisition device is a stereoscopic image acquisition device, wherein the stereoscopic image acquisition device records the intraoperative image data as stereoscopic intraoperative image data including depth information about its image points, wherein the depth information is used to calculate the first intraoperative point cloud and / or the second intraoperative point cloud.

[0031] In particular, the intraoperative image acquisition device is a laparoscope or endoscope. Surgeons use the intraoperative image acquisition device to capture image data during surgery. By using a stereoscopic image acquisition device, depth information about image points can be obtained, which is used to calculate the point cloud.

[0032] Preferably, the processing unit receives multiple sets of intraoperative image data captured at different time points, wherein the processing unit detects the local deformation motion in the intraoperative image data sets by comparing the sets of intraoperative image data, wherein the processing unit classifies the intraoperative image data sets captured before the local deformation motion as the first intraoperative image data, and classifies the intraoperative image data sets captured after the local deformation motion as the second intraoperative image data.

[0033] The image data received by the processing unit can be an intraoperative image data stream captured by the intraoperative image acquisition device. From the intraoperative image data stream, images captured before local deformation movement are designated as first intraoperative image data, and images captured after local deformation movement are designated as second intraoperative image data. In particular, all sets of intraoperative image data are captured from the same viewpoint. In other words, the intraoperative image acquisition device does not move, rotate, or tilt in any way when capturing intraoperative image data.

[0034] Preferably, the processing unit uses optical flow and / or optical expansion to detect the local deformation motion in the intraoperative image dataset.

[0035] Optical flow and / or optical dilation methods can be used to detect moving parts in intraoperative laparoscopic views. These methods are known in the art and are described, for example, in “G. Yang, D. Ramanan, Upgrading Optical Flow to 3DScene Flow through Optical Expansion, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 1334-1343”. In particular, optical flow and / or optical dilation can be used to classify image points into moving points and / or stationary points.

[0036] Preferably, the processing unit receives multiple sets of intraoperative image data captured at different time points, wherein the processing unit receives a time signal indicating the completion of the local deformation movement, wherein the processing unit classifies the intraoperative image data set captured after receiving the time signal as the second intraoperative image data, wherein, in particular, the processing unit classifies the intraoperative image data set captured before receiving the time signal as the first intraoperative image data.

[0037] A time signal can be generated by pressing a button. In this way, the surgeon can indicate the completion of the migration step (i.e., local deformation movement). The generation and transmission of the time signal in this manner provides a quick and simple way to indicate a first time point and a second time point. Specifically, the processing unit receives the second time signal before the local deformation movement begins, wherein the processing unit classifies the intraoperative image data set captured before receiving the second time signal as the first intraoperative image data.

[0038] Preferably, the geometric feature vector is computed and assigned to the at least one fixed point using a feature descriptor program (especially a fast point feature histogram descriptor program).

[0039] Feature descriptor programs (also known as feature descriptors) compute geometric feature vectors and assign them to image points. In particular, feature descriptor programs are 3D point cloud descriptors. An example of a 3D point cloud descriptor is the Fast Point Feature Histogram descriptor, known from “RBRusu, N. Blodow and M. Beetz, Fast Point Feature Histograms (FPFH) for 3D registration, 2009 IEEE International Conference on Robotics and Automation”. Other types of 3D point cloud descriptors include, for example, the Signature of Histogram of Orientation (SHOT) descriptor or the Spin Image descriptor. An overview of the different types of 3D point cloud descriptors can be found in “XFHan, JSJin, J.Xie, MJWang, W.Jiang, A comprehensive review of 3D point cloud descriptors, arXiv:1802.02297v1 [cs.CV] 7 Feb 2018”.

[0040] According to an implementation, the positional change of the at least one neighboring moving point and / or the local deformation of the organ are estimated using an artificial neural network (especially a deep learning network).

[0041] Artificial neural networks are trained using characteristic difference vectors generated by typical local deformation motions. In this way, artificial neural networks learn to recognize local deformation motions quickly and reliably. The use of artificial neural networks greatly simplifies the task of estimating the positional changes of at least one neighboring moving point and / or the local deformation of an organ.

[0042] Preferably, the artificial neural network is trained to identify patterns of difference vectors generated by the local deformational movements of the organ at multiple fixed points.

[0043] From these patterns, artificial neural networks infer local deformation motions. These patterns can be obtained through histogram analysis of the difference vectors. By analyzing the patterns in the difference vectors, the deformation modeling task is decomposed into a pattern recognition task that is well-suited for solving by artificial neural networks.

[0044] This objective is further achieved through a method for simulating local deformations of a three-dimensional model of an organ using a medical image processing system, wherein the local deformations are estimated using a method of operating a medical imaging system according to one of the foregoing embodiments, wherein the medical image processing system includes a storage unit storing the model, the model being derived from preoperative image data of the organ in an undeformed state, wherein, after estimating the local deformations of the organ, the processing unit:

[0045] - A non-rigid transformation matrix is ​​estimated using the positional changes of the at least one neighboring moving point, wherein the non-rigid transformation matrix, when applied to the model, reproduces the local deformation motion of the organ.

[0046] - Apply the non-rigid transformation matrix to the model to correct the model for the local deformation of the organ.

[0047] The same or similar advantages mentioned previously regarding methods for operating medical imaging systems are applicable to methods for simulating local deformations of three-dimensional models.

[0048] By utilizing a method that simulates local deformations in a 3D model, visible local deformations from intraoperative images can be applied to the model, allowing it to be overlaid onto the intraoperative images to enrich it with additional information. The use of a standardized transfer step simplifies the estimation of the non-rigid transformation matrix because the number of local deformations is finite. In particular, the model is calculated based on preoperative CT and / or MR image data. Specifically, the model is a preoperative point cloud.

[0049] Preferably, each image point in the model is associated with a corresponding image point in the second intraoperative point cloud. In particular, each image point in the model is associated with a corresponding image point in the second intraoperative point cloud by assigning geometric feature vectors to the image points in the model and the image points in the second intraoperative point cloud, and registering the second intraoperative point cloud to the model based on the similarity of the corresponding geometric feature vectors.

[0050] By associating image points in the model with corresponding image points in the second intraoperative point cloud, the estimated local deformations in the intraoperative point cloud can be easily transferred to the model. Applying geometric feature vectors to image points, especially in combination with global registration algorithms, is a fast and effective method for establishing point correspondences between the preoperative model and one or more intraoperative point clouds.

[0051] Preferably, the non-rigid transformation matrix is ​​estimated via an artificial neural network (especially a deep learning network).

[0052] In particular, the artificial neural network is trained to estimate the non-rigid transformation matrix directly from the difference vector of at least one fixed point. In doing so, the artificial neural network also estimates the positional changes of neighboring moving points and the local deformation of the organ. In other words, the tasks of estimating the positional changes of neighboring moving points, the local deformation of the organ, and the non-rigid transformation matrix are performed by the same artificial neural network. By computing the non-rigid transformation matrix, the neural network simultaneously estimates both positional changes and local deformation motions.

[0053] Preferably, the artificial neural network is trained using a dataset that includes non-rigid transformation matrices corresponding to different local deformation motions of the organ and difference vectors corresponding to these local deformation motions.

[0054] By training artificial neural networks in this way, they learn to recognize patterns associated with different local deformation movements or migration steps.

[0055] This objective is also achieved by a medical image processing system comprising a processing unit, wherein the medical image processing system is designed and configured to perform a method for operating a medical imaging system according to any of the foregoing embodiments and / or a method for local deformation of a three-dimensional model of a simulated organ according to any of the foregoing embodiments.

[0056] The same or similar advantages mentioned previously regarding methods for simulating local deformation of 3D models and methods for operating medical imaging systems are applicable to medical image processing systems.

[0057] This objective is also achieved by a software program product comprising program code means for a medical image processing system according to any of the foregoing embodiments, the software program product comprising a control program component that executes in the processing unit of the medical image processing system, characterized in that the control program component is designed to execute the method according to any of the foregoing embodiments when executed in the processing unit.

[0058] The same or similar advantages mentioned previously regarding medical image processing systems, methods for simulating local deformation of 3D models, and methods for operating medical imaging systems are applicable to software program products. Attached Figure Description

[0059] Further features of the invention will become apparent from the description of embodiments according to the invention, as well as from the claims and the accompanying drawings. Individual features or combinations of features can be implemented according to embodiments of the invention.

[0060] The invention is described below based on exemplary embodiments, without limiting the overall intent of the invention, wherein disclosure of all details of the invention not explained in more detail in the text is expressly referenced to the accompanying drawings. The drawings are shown below:

[0061] Figure 1 A simplified schematic diagram of a medical image processing system is shown.

[0062] Figure 2 This is a simplified schematic diagram illustrating intraoperative images of the liver before local deformation and movement.

[0063] Figure 3 A simplified schematic diagram illustrating intraoperative images of the liver after localized deformable movement is shown.

[0064] Figure 4 This is a simplified schematic diagram of several points in a point cloud, representing the liver before local deformation and movement.

[0065] Figure 5 This is a simplified schematic diagram of several points in a point cloud, representing the liver after local deformation.

[0066] Figure 6 Examples from Figure 4 and Figure 5 A simplified diagram illustrating the changes in the positions of several points within a given point and the surface normal.

[0067] Figures 7a to 7c Examples are given indicating that the source is Figure 4 and Figure 5 The histogram of the geometric eigenvectors of the first fixed point at the first time point ( Figure 7a Histogram of geometric eigenvectors at the second time point ( ) Figure 7b ) and the histogram of the difference vectors ( Figure 7c ),

[0068] Figures 8a to 8c Examples are given indicating that the source is Figure 4 and Figure 5 Histogram of the geometric eigenvectors of the second fixed point at the first time point ( Figure 8a Histogram of geometric eigenvectors at the second time point ( ) Figure 8b ) and the histogram of the difference vectors ( Figure 8c ),

[0069] Figures 9a to 9c Examples are given indicating that the source is Figure 4 and Figure 5 The histogram of the geometric eigenvectors of the third fixed point at the first time point ( Figure 9a Histogram of geometric eigenvectors at the second time point ( ) Figure 9b ) and the histogram of the difference vectors ( Figure 9c ),

[0070] In the accompanying drawings, elements of the same or similar types, or corresponding parts respectively, are given the same reference numerals to avoid the need to reintroduce the item.

[0071] List of reference numerals

[0072] 2 Medical Imaging System

[0073] 4 Processing Units

[0074] 5. Software Program Products

[0075] 6 storage units

[0076] 7 Models

[0077] 8. Intraoperative image acquisition device

[0078] 9. Tracking System

[0079] 10 organs

[0080] 12 Organ Surface

[0081] 14 Right side

[0082] 16 Left side

[0083] 17. Falciform ligament

[0084] 18 Local Deformation Motion

[0085] 19 Markers

[0086] Fixed points 20, 21, and 22

[0087] 23, 24, 25 Predefined spaces

[0088] 30, 31, 32 Neighboring moving points

[0089] 35 Surface Normal

[0090] 40 Geometric eigenvectors

[0091] 42 Difference Vectors

[0092] Values ​​of vector components 44, 45, 46, and 47 Detailed Implementation

[0093] Figure 1A schematic diagram of an exemplary embodiment of a medical imaging system 2 is shown, which is designed and configured to estimate local deformation of organ 10. System 2 includes a processing unit 4 such as a CPU or similar device, a storage unit 6, an intraoperative image acquisition device 8, and a tracking system 9. The storage unit 6 is designed and configured to store data. A three-dimensional model 7 derived from preoperative image data is stored on the storage unit 6. The intraoperative image acquisition device 8 is, for example, a laparoscope or endoscope.

[0094] The software program product 5 stored on the storage unit 6 and executed by the processing unit 4 can be used to perform methods for operating the medical imaging system 2 to estimate local deformation of the organ 10 and / or methods for simulating local deformation of the three-dimensional model 7 of the organ 10, as described below.

[0095] The intraoperative image acquisition device 8 captures intraoperative image data during the intraoperative procedure and transmits it to the processing unit 4. The processing unit 4 calculates a point cloud based on individual intraoperative images or an intraoperative image stream including multiple intraoperative images.

[0096] To receive depth information about image points required for calculating the point cloud, according to the illustrated embodiment, a tracking system 9 is used to track the position of a marker 19 placed on the organ 10. The marker 19 may be an electronic and / or optical marker that allows the tracking system 9 to determine the position of the marker 19 in three-dimensional space. The tracking system 9 transmits this information to the processing unit 4. Alternatively, the depth information is acquired by a stereoscopic intraoperative image acquisition device 8.

[0097] Model 7 (or a point cloud) is aligned and overlaid onto the intraoperative image data to enrich the intraoperative images with additional information. This requires that the organ 10 represented by the model matches the organ 10 in the intraoperative images. However, this may not be the case due to local deformation of the organ during the surgical procedure, which will... Figure 2 and Figure 3 The explanation is provided below.

[0098] Figure 2 A schematic diagram of an organ, in this case the liver, is shown, having an organ surface 12. The liver comprises a right side 14 and a left side 16 separated by a falciform ligament 17. During laparoscopic surgery, due to the limited space in the abdominal cavity, migration techniques are required to move portions of the organ 10 to suitable locations for resection. For example, the left side 16 of the liver is folded up to better access previously unseen portions of the liver.

[0099] exist Figure 3 The diagram schematically illustrates this local deformation movement 18 involving the upward movement of the left-hand portion 16. However, Figure 3 The movement 18 shown is purely illustrative and is not intended to be limited to any specific migration step performed by a surgeon. Figure 3 As shown, due to the local deformation motion 18, the position of the left side 16 changes, while the right side 14 remains stationary due to the falciform ligament 17 acting as the tilt axis.

[0100] During the migration step performed during the surgery, organ 10 will be locally deformed. In order to align and overlay the preoperative model 7 depicting the undeformed organ 10 onto the intraoperative image data depicting the deformed organ 10, model 7 must be corrected for the local deformation. To do this, the local deformation motion must be identified and appropriate corrections applied to model 7.

[0101] The following is combined Figures 4 to 9c This describes a method for operating the medical imaging system 2 to estimate local deformation of organ 10. Figure 4 A first intraoperative point cloud depicting the surface of organ 10 in its undeformed state is shown. This first intraoperative point cloud is calculated by processing unit 4 based on intraoperative image data captured by intraoperative image acquisition device 8 before local deformation movement. Multiple points 20, 21, and 22 of the point cloud are indicated on the right side 14 of organ 10, and multiple points 30, 31, and 32 are indicated on the left side 16 of organ 10. Due to local deformation movement 18, points 30, 31, and 32 will move upwards, while the spatial positions of points 20, 21, and 22 will remain unchanged, as shown below. Figure 5 As shown. Therefore, points 30, 31, and 32 are classified as moving points 30, 31, and 32, while points 20, 21, and 22 are classified as stationary points 20, 21, and 22.

[0102] The feature descriptor procedure assigns geometric feature vectors to fixed points 20, 21, and 22. These geometric feature vectors indicate the different geometric relationships between fixed points 20, 21, and 22 and their neighboring points. For example, a point's geometric relationship could be the orientation of its surface normal, the angle between its surface normal and the surface normals of neighboring points, or its distance to neighboring points in 3D space. The neighboring points of any given point are defined as those points located within a predefined space 23, 24, and 25 centered on that given point. Figure 4 and Figure 5 In the context, predefined spaces 23, 24, and 25 are circles or spheres with a certain radius centered on fixed points 20, 21, and 22. The radius can be changed as needed to alter the sensitivity of fixed points 20, 21, and 22 to their surrounding environment.

[0103] To estimate the local deformation motion 18, the geometric eigenvectors assigned to fixed points 20, 21, and 22 are... Figure 4 The first time point shown is Figure 5The changes between the second time points shown were calculated. Although the spatial positions of fixed points 20, 21, and 22 remain unchanged, their geometric eigenvectors change whenever the position of at least one of their neighboring points changes. From Figure 4 and Figure 5 As can be seen, the predefined space 23 of fixed point 20, represented by the dashed line, includes fixed points 21 and 22 and moving point 30. The predefined space 24 of fixed point 21, represented by the dotted line, includes fixed points 20 and 22 and moving points 30 and 31. The predefined space 25 of fixed point 22, represented by the dashed line, includes fixed points 20 and 21 and all moving points 30, 31, and 32.

[0104] Figure 6 The diagram schematically illustrates the changes in the positions of moving points 30, 31, and 32 and their surface normals 35 caused by the local deformation motion 18. It can be seen that the positions and surface normals 35 of the fixed points 20, 21, and 22 remain unchanged. At the first time point, moving points 30, 31, and 32 are positioned in the same plane as the fixed points 20, 21, and 22, indicated by dashed lines. However, at the second time point, indicated by solid lines, moving points 30, 31, and 32 move upwards. Simultaneously, the surface normals 35 of moving points 30, 31, and 32 tilt to the left due to the local deformation motion 18.

[0105] like Figures 7a to 9c As shown, the geometric eigenvectors of the fixed points indicate the positions of the moving points 30, 31, and 32 and these changes in their surface normals 35. Figure 7a The geometric eigenvector 40 of fixed point 20 is shown at a first time point prior to the local deformation motion. The geometric eigenvector 40 is represented as a histogram. The individual markers on the horizontal axis of the histogram represent another vector component of the geometric eigenvector 40, and thus represent another geometric relation indicated by the eigenvector 40. The vertical axis represents the number of neighboring points to which these geometric relations apply. Figure 4 and Figure 5 It can be seen that the predefined space 23 of fixed point 20 includes points 21, 22, and 30, a total of three points. Since the directions of the surface normal 35 of the three points 21, 22, and 30 are the same at the first time point, as... Figure 6 As shown, the value 44 of the vector component indicating this geometric relationship is therefore in Figure 7a The value in the middle is 3.

[0106] Figure 7b The geometric characteristic vector 40 of the fixed point 20 at the second time point is depicted. As the surface normal 35 of the moving point 30 changes due to the local deformation motion 18, the value 44 of the vector component indicating the perpendicular surface normal changes to 2, while the value 45 of the vector component indicating the inclined surface normal 35 changes to 1.

[0107] Figure 7c The difference vector 42 of fixed point 20 is depicted. The difference vector 42 is calculated by subtracting the geometric eigenvector 40 at the first time point from the geometric eigenvector 40 at the second time point, and vice versa. Therefore, the difference vector 42 indicates the change of the geometric eigenvector 40, which is indicated by the values ​​of the vector components 46 and 47.

[0108] Figures 8a to 8c The geometric eigenvectors 40 and 42 of fixed point 21 at two time points are shown. Since the predefined space 24 of fixed point 21 comprises a total of four points, the values ​​of the vector components 44, 45, 46, and 47 change accordingly. Similarly, Figures 9a to 9c The geometric eigenvectors 40 and difference vectors 42 of fixed point 22 at two time points are shown, and its predefined space 25 includes a total of five points.

[0109] The choice of three moving points 30, 31, and 32 and three fixed points 20, 21, and 22 is solely for better representability. The number of moving points 30, 31, and 32 and fixed points 20, 21, and 22 can be less than three or much more than three. The geometric eigenvectors assigned to fixed points 20, 21, and 22 can include any number of vector components. For example... Figures 7a to 9c The number of the three vector components shown is merely exemplary. According to embodiments, the geometric feature vector may include at least ten, especially at least forty, especially at least one hundred vector components, each vector component indicating a different geometric relationship.

[0110] By calculating the difference vector 42 between fixed points 20, 21, and 22, the positional changes of moving points 30, 31, and 32 are estimated, thereby estimating the local deformation motion. This knowledge of the local deformation motion can be used to calculate the non-rigid transformation matrix. When applied to model 7, the non-rigid transformation matrix reproduces the local deformation motion of organ 10. By applying the non-rigid transformation matrix to model 7, model 7 is corrected for the local deformation of organ 10.

[0111] According to the implementation, an artificial neural network (e.g., a deep learning network) is trained using the typical pattern of the difference vector 42 of the fixed points 20, 21, and 22 generated by typical migration movements. The artificial neural network can also estimate non-rigid transformation matrices. The estimation or computation of the matrix can be performed directly from the difference vector 42. In this case, the estimation of the positional changes and local deformation movements of the moving points is achieved by estimating the matrix. In particular, the artificial neural network is stored in the storage unit 6 and executed by the processing unit 4.

[0112] All named features, including those individually derived from the accompanying drawings and those disclosed in combination with other features, are considered individually and in combination as essential to the invention. Embodiments according to the invention can be achieved by individual features or combinations of features. Features combined with the wording "especially" or "particularly" are considered preferred embodiments.

Claims

1. A method for operating a medical imaging system (2) to estimate local deformation of an organ (10), wherein, The medical imaging system (2) includes a processing unit (4), wherein the processing unit (4) - Receive first intraoperative image data of the organ (10) captured at a first time point and second intraoperative image data of the organ (10) captured at a second time point, wherein the first time point is before the local deformation movement (18) of the organ (10) and the second time point is after the local deformation movement (18) of the organ (10). - A first intraoperative point cloud is calculated based on the first intraoperative image data and a second intraoperative point cloud is calculated based on the second intraoperative image data, wherein the first intraoperative point cloud includes information about the spatial location of image points representing the organ surface (12) at the first time point, and wherein the second intraoperative point cloud includes information about the spatial location of the image points representing the organ surface (12) at the second time point. - At least one image point in the first intraoperative point cloud and its corresponding point in the second intraoperative point cloud are classified as fixed points (20, 21, 22), wherein the spatial location of the fixed points (20, 21, 22) is the same at the first time point and at the second time point. - Calculate a geometric feature vector (40) and assign it to at least one fixed point (20, 21, 22) in the first intraoperative point cloud and the second intraoperative point cloud, wherein the geometric feature vector (40) of the fixed point (20, 21, 22) indicates at least one geometric relationship between the fixed point (20, 21, 22) and at least one of its neighboring image points. - Calculate the difference vector (42) of the at least one fixed point (20, 21, 22), wherein the difference vector (42) is the difference between the geometric feature vector (40) at the first time point and the geometric feature vector (40) at the second time point. - At least one image point in the first intraoperative image data is designated as a neighboring moving point (30, 31, 32), wherein the neighboring moving points (30, 31, 32) are located in a predefined space (23, 24, 25) centered on the at least one fixed point (20, 21, 22), and wherein the spatial position of the neighboring moving points (30, 31, 32) changes between the first time point and the second time point. - The positional change of at least one neighboring moving point (30, 31, 32) is estimated based on the difference vector (42) of at least one fixed point (20, 21, 22), and the positional change is used to estimate the local deformation of the organ (10).

2. The method according to claim 1, wherein, In order to calculate the first intraoperative point cloud and / or the second intraoperative point cloud, optical and / or electronic markers (19) are placed on the organ surface (12), and the position of the markers (19) is determined using a tracking system (9), wherein the processing unit (4) uses the position of the markers (19) to calculate the position of the image points of the first intraoperative point cloud and / or the position of the image points of the second intraoperative point cloud.

3. The method according to claim 1, wherein, The medical imaging system (2) includes an intraoperative image acquisition device (8), which records intraoperative image data and transmits the intraoperative image data to the processing unit (4).

4. The method according to claim 3, wherein, The intraoperative image data is the first intraoperative image data and / or the second intraoperative image data.

5. The method according to claim 3, wherein, The intraoperative image acquisition device (8) is a stereoscopic image acquisition device, wherein the stereoscopic image acquisition device records the intraoperative image data as stereoscopic intraoperative image data including depth information about its image points, wherein the depth information is used to calculate the first intraoperative point cloud and / or the second intraoperative point cloud.

6. The method according to claim 1, wherein, The processing unit (4) receives multiple sets of intraoperative image data captured at different time points, wherein the processing unit (4) detects the local deformation motion (18) in the intraoperative image data set by comparing the sets of intraoperative image data, wherein the processing unit classifies the intraoperative image data set captured before the local deformation motion as the first intraoperative image data, and classifies the intraoperative image data set captured after the local deformation motion as the second intraoperative image data.

7. The method according to claim 6, wherein, The processing unit (4) uses optical flow and / or optical dilatation to detect the local deformation motion (18) in the intraoperative image dataset.

8. The method according to claim 1, wherein, The processing unit (4) receives multiple sets of intraoperative image data captured at different time points, wherein the processing unit receives a time signal indicating the completion of the local deformation movement (18), and wherein the processing unit classifies the sets of intraoperative image data captured after receiving the time signal as the second intraoperative image data.

9. The method according to claim 8, wherein, The processing unit will classify the set of intraoperative image data captured before receiving the time signal as the first intraoperative image data.

10. The method according to claim 1, wherein, The geometric feature vector (40) is computed using a feature descriptor program and assigned to the at least one fixed point (20, 21, 22).

11. The method according to claim 10, wherein, The feature descriptor program is a fast point feature histogram descriptor program.

12. The method according to claim 1, wherein, The positional changes of the at least one neighboring moving point (30, 31, 32) and / or the local deformation of the organ (10) are estimated using an artificial neural network.

13. The method according to claim 12, wherein, The artificial neural network in question is a deep learning network.

14. The method according to claim 12, wherein, The artificial neural network is trained to identify patterns of the difference vectors (42) generated by the local deformation motions (18) of the organ (10) at multiple fixed points (20, 21, 22).

15. A method for simulating local deformation of a three-dimensional model (7) of an organ (10) using a medical imaging system (2), wherein, The local deformation is estimated using a method of estimating the local deformation of the organ (10) using an operational medical imaging system (2) according to any one of claims 1 to 14, wherein the medical imaging system (2) includes a storage unit (6) storing the model (7) derived from preoperative image data of the organ (10) in an undeformed state, wherein, after estimating the local deformation of the organ (10), the processing unit (4): - The non-rigid transformation matrix is ​​estimated by utilizing the positional changes of at least one neighboring moving point (30, 31, 32), wherein the non-rigid transformation matrix reproduces the local deformation motion (18) of the organ (10) when applied to the model (7). - The non-rigid transformation matrix is ​​applied to the model (7) to correct the model (7) for the local deformation of the organ (10).

16. The method according to claim 15, wherein, Each image point in the model (7) is associated with a corresponding image point in the second intraoperative point cloud.

17. The method according to claim 16, wherein, Each image point in the model (7) is associated with a corresponding image point in the second intraoperative point cloud by assigning a geometric feature vector (40) to the image point in the model (7) and the image point in the second intraoperative point cloud, and registering the second intraoperative point cloud to the model (7) based on the similarity of the corresponding geometric feature vectors (40).

18. The method according to claim 15, wherein, The non-rigid transformation matrix is ​​estimated via an artificial neural network.

19. The method according to claim 18, wherein, The artificial neural network in question is a deep learning network.

20. The method according to claim 18, wherein, The artificial neural network is trained using a dataset that includes non-rigid transformation matrices corresponding to different local deformation movements (18) of the organ and difference vectors (42) corresponding to these local deformation movements (18).

21. A medical imaging system (2), the medical imaging system (2) comprising a processing unit (4), wherein, The medical imaging system (2) is designed and configured to perform a method of operating the medical imaging system (2) according to any one of claims 1 to 14 to estimate local deformation of the organ (10) and / or a method of using the medical imaging system (2) according to any one of claims 15 to 20 to simulate local deformation of a three-dimensional model (7) of the organ (10).

22. A software program product (5) comprising program code means for a medical imaging system (2) according to claim 21, said software program product (5) comprising a control program component executed in the processing unit (4) of said medical imaging system (2), characterized in that, The control program component is designed to perform the method according to any one of claims 1 to 20 when executed in the processing unit (4).

Images