Preoperative and intraoperative liver point cloud data registration system, method, terminal and storage medium
By extracting and fusing local and global features of liver point clouds from preoperative and intraoperative data using deep learning methods, and filtering them using overlapping region masks, the problem of insufficient registration accuracy of preoperative and intraoperative data in 3D laparoscopic liver tumor resection was solved, achieving higher accuracy in liver tumor localization and resection.
Patent Information
- Application Number
- CN202211197211.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-29
- Publication Date
- 2026-05-15
- Estimated Expiration
- 2042-09-29
AI Technical Summary
Existing medical image registration techniques struggle to achieve precise alignment between preoperative liver CT images and intraoperative liver surface data under laparoscopic visualization during 3D laparoscopic liver tumor resection. This is particularly true for low-overlap data, where the registration effect is poor, impacting surgical accuracy and safety.
The system employs a local hybrid feature extraction module, a global feature extraction module, a feature fusion module, an overlapping region mask prediction module, and a transformation matrix prediction module. It automatically extracts local and global features from preoperative and intraoperative point cloud data using deep learning methods, filters them using the learned overlapping region mask, and calculates the spatial transformation matrix for registration.
It improves the registration accuracy of liver point cloud data before and during surgery, avoids local optimum errors, adapts to liver point cloud data registration under 3D laparoscopy, and improves the accuracy and safety of surgery.
Smart Images
Figure CN115527003B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a preoperative and intraoperative liver point cloud data registration system, method, terminal, and storage medium. Background Technology
[0002] The key to 3D laparoscopic liver tumor resection lies in the precise localization of liver tumors under 3D laparoscopic visualization. Medical image registration technology is an effective method to solve this problem. Registration refers to the precise alignment of the patient's preoperative or intraoperative images (image space) with the anatomical structures (physical space) of the surgical area. Medical image registration technology typically uses both 2D and 3D data: 2D medical images can lead to some loss of information about the affected area; while 3D data can display more intuitive and detailed pathological information. Using 3D data for registration can improve the accuracy and stability of diagnosis and surgical planning. In 3D laparoscopic liver tumor resection, the images used for registration are preoperative liver CT images and intraoperative liver surface data under laparoscopic visualization. Using registration technology enables precise localization of liver tumors, improving surgical precision and safety. Less experienced surgeons can use this technology to perform liver resection surgery more safely, while skilled professionals can use it to improve surgical accuracy, shorten operation time, increase tumor resection rate, and reduce tumor residue and local recurrence rates.
[0003] Traditional marker-based medical image registration techniques require manual selection of markers. This is difficult to apply to 3D laparoscopic liver tumor resection, where it's challenging to set markers in the surgical area during the procedure. Because the liver surface is relatively smooth and lacks representative feature points, feature-point matching-based medical image registration techniques struggle to obtain sufficient registration information on the liver surface, resulting in registration accuracy insufficient for 3D laparoscopic liver tumor resection. Deep learning-based point cloud registration techniques can acquire latent features from point cloud data, achieving higher registration accuracy than traditional methods. However, the registration effect is often limited by the training dataset used. Due to the limited number of labeled medical image datasets, current training sets typically consist of data from natural scenes, which are not well-suited for medical image registration. Furthermore, the narrow field of view in 3D laparoscopy only shows a portion of the liver surface, and the liver undergoes significant deformation due to pressure from pneumoperitoneum and variations in patient respiration and position. Preoperative and intraoperative point cloud data often only have local overlap, and current deep learning registration techniques do not yet meet surgical requirements for low-overlap data.
[0004] Reference [1] "Rpm-net: Robust point matching using learned features" proposes RPMNet based on the robust point matching (RPM) algorithm. In the feature extraction part, a structure similar to PPFNet is adopted, and the three-dimensional coordinates of the points are added to the features to obtain mixed features; a parameter prediction network is used to estimate the outlier parameters and annealing parameters, and then the mixed features are combined to calculate the point pair matching. In the point pair outlier removal stage, a differentiable sinkhorn layer is introduced to expand the matching matrix, and then iterative normalization is performed to obtain a correspondence with higher confidence. RPMNet achieves excellent performance under noise and partial overlap, but this method requires repeated calculation of features in the iteration, which has a high computational cost.
[0005] Reference [2] "Omnet: Learning overlapping mask for partial-to-partial pointcloud registration" proposes OMNet (overlapping mask network). This method, in order to avoid the negative impact of non-overlapping points, predicts the overlapping mask of the two source point clouds and the target point cloud in each iteration, filters the non-overlapping regions, and then predicts the relative motion parameters from the global features of the two point clouds through multilayer perceptrons (MLPs). OMNet avoids interference with global features by removing non-overlapping regions and has achieved excellent results on low-overlap point cloud data. However, the extracted global features of the point cloud contain less local information and are not suitable for liver surface data. Summary of the Invention
[0006] In view of this, it is necessary to provide a method for registering preoperative and intraoperative liver point cloud data, which can register preoperative liver CT image data and intraoperative liver surface data under laparoscopic vision for low overlap data.
[0007] This invention provides a preoperative and intraoperative liver point cloud data registration system, comprising: a local hybrid feature extraction module for extracting local hybrid features containing geometric structure information from preoperative and intraoperative point cloud data respectively; a global feature extraction module for fusing the extracted local hybrid features to obtain global features of the preoperative point cloud and global features of the intraoperative point cloud; and a feature fusion module for fusing the local hybrid features, global features, and global features of the preoperative point cloud to obtain fused features of the preoperative point cloud; similarly, the local hybrid features, global features, and global features of the intraoperative point cloud are fused to obtain fused features of the preoperative point cloud; The system integrates global features to obtain the integrated features of the intraoperative point cloud. An overlapping region mask prediction module integrates the integrated features of the preoperative and intraoperative point clouds to obtain their respective overlapping region masks and decoding features. A transformation matrix prediction module multiplies the overlapping region masks of the preoperative and intraoperative point clouds with their respective local blending features and links them with their respective decoding features to obtain the spatial transformation matrices of the preoperative and intraoperative point clouds. A registration module applies the calculated spatial transformation matrices to the preoperative point cloud data to obtain the registration results between the preoperative and intraoperative point cloud data.
[0008] Preferably, the local mixing feature f X It consists of three parts: the spatial coordinates of the point, the distance to neighboring points, and local geometric features; for a point X in the point cloud X... i Let its neighborhood be N(X) i If the local mixing characteristics are then... Represented as:
[0009]
[0010] Where f(·) represents a multilayer perceptron network used to extract local blending features of the point cloud, X j ∈N(X i ), This represents the overlapping region mask obtained in the previous iteration. This means converting neighborhood points into local neighbors after subtracting the centroid point:
[0011]
[0012] PPF(X i ,X j ) represents X j With X i The 4D point-pair spatial feature (PPF) between two spatial points is described using the distance between the two spatial points and the angle between their normal vectors.
[0013]
[0014] Where, n i and n j Representing point X i With X j The normal vector.
[0015] Preferably, the global feature extraction module is specifically used for:
[0016] The local blending features of all points in the point cloud X in the i-th iteration are extracted. Then, the local mixed feature set is fed into a three-layer convolutional network, expanded to 1024-D, and then combined with the overlapping region mask obtained from the previous iteration. Multiply the results and pass them through a max-pooling layer to obtain the final global features.
[0017] Preferably, the feature fusion module is specifically used for:
[0018] The feature fusion module contains three convolutional layers, which take local mixed features of the preoperative point cloud X as input. Global features and global features of intraoperative point cloud Y Output fusion features of 512-dimensional preoperative point cloud Similarly, the fusion features of the intraoperative point cloud are obtained.
[0019] Preferably, the overlapping region mask prediction module is specifically used for:
[0020] Global features of point cloud X and Y and and point-by-point features and The concatenation and interactive feedback in each iteration continuously improve the local mixed features. and After performing adjustments and transformations, and then fusing the above features, a new overlapping region mask is obtained. and regression features used to calculate the transformation matrix
[0021]
[0022]
[0023] Where g(·) represents the feature fusion module, f(·) represents the overlapping region mask prediction module, and the input fused features are... After four convolutional layers, the output is the predicted overlapping region mask. The decoded features of the point cloud are obtained by concatenating the outputs of the first three layers.
[0024] Preferably, the transformation matrix prediction module is specifically used for:
[0025] The transformation matrix prediction module contains 5 convolutional layers. The input features are fed into the transformation matrix prediction module after a single max pooling operation. After 5 convolutional layers, a 7-d feature vector is output, representing the spatial transformation of the current iteration, where the first 4 values represent the rotation quaternions q∈R. 4 The last three values represent the translation vector t∈R 3 The input feature refers to the feature formed by combining the fused feature and the layer-by-layer concatenation feature;
[0026]
[0027] in, and p(·) represents the overlapping region mask predicted in the previous iteration, and p(·) represents the transformation matrix prediction module. After the iteration ends, the spatial transformations predicted in each round will be accumulated and calculated to obtain the final overall transformation between the preoperative point cloud and the intraoperative point cloud.
[0028] Preferably, the registration module is specifically used for:
[0029] Once the spatial transformation matrix T between the preoperative and intraoperative point clouds is determined, the transformation process can begin. Specifically, the virtual position of the preoperative point cloud within the intraoperative point cloud region is denoted as P. T The original location of the preoperative point cloud is denoted as P. I Then we have:
[0030] P T =T·P I .
[0031] This invention also provides a method for registering preoperative and intraoperative liver point cloud data, comprising the following steps: a. extracting local hybrid features containing geometric structure information from preoperative and intraoperative point cloud data respectively; b. fusing the extracted local hybrid features to obtain global features of the preoperative point cloud and global features of the intraoperative point cloud; c. fusing the local hybrid features, global features, and global features of the preoperative point cloud to obtain fused features of the preoperative point cloud; similarly, fusing the local hybrid features, global features, and global features of the intraoperative point cloud to obtain fused features of the preoperative point cloud; d. The global features of the point cloud are fused to obtain the fused features of the intraoperative point cloud; e. The fused features of the preoperative point cloud and the intraoperative point cloud are fused to obtain their respective overlapping region masks and decoding features; f. The overlapping region masks of the preoperative point cloud and the intraoperative point cloud are multiplied by their respective local blending features and linked with their respective decoding features to obtain the spatial transformation matrix of the preoperative point cloud and the intraoperative point cloud; f. The calculated spatial transformation matrix is applied to the preoperative point cloud data to obtain the registration result of the preoperative point cloud data and the intraoperative point cloud data.
[0032] The present invention also provides a terminal, the terminal including a processor and a memory coupled to the processor, wherein: the memory stores program instructions for implementing the preoperative and intraoperative liver point cloud data registration method; the processor is used to execute the program instructions stored in the memory to implement preoperative and intraoperative liver point cloud data registration.
[0033] The present invention also provides a storage medium storing processor-executable program instructions for executing the preoperative and intraoperative liver point cloud data registration method.
[0034] The beneficial effects of this application include: the features of preoperative and intraoperative point clouds are automatically extracted using deep learning methods and obtained by fusing local and global features. Local features mainly focus on the local set information of the point cloud, while global features have a broader field of view and can focus on the overall structure of the point cloud. Combining the two can effectively improve the registration accuracy of the point cloud and avoid getting trapped in the erroneous state of local optima; the point cloud is filtered by using the learned point cloud overlapping region mask to reject points in non-overlapping regions, and the local-to-local point cloud registration is converted into point cloud registration of the same shape, which can effectively improve the registration accuracy of preoperative liver point cloud data and intraoperative laparoscopic liver surface point cloud. Attached Figure Description
[0035] Figure 1 This is a schematic diagram of the preoperative and intraoperative liver point cloud data registration system according to an embodiment of this application;
[0036] Figure 2 This is a flowchart of the preoperative and intraoperative liver point cloud data registration method according to an embodiment of this application;
[0037] Figure 3 This is a schematic diagram of the terminal structure according to an embodiment of this application;
[0038] Figure 4 This is a schematic diagram of the structure of the storage medium according to an embodiment of this application;
[0039] Figure 5 A schematic diagram of the registration visualization results of this application, reference [1], and reference [2] provided for the embodiments of this application: (a) is this application, (b) is reference [1], (c) is reference [2], and (d) is the registration true value of two point clouds; wherein, the point cloud with the smallest diameter and lowest brightness represents the preoperative point cloud, the point cloud with the largest diameter and highest brightness represents the intraoperative point cloud, and the point cloud with the diameter and brightness in the middle represents the preoperative point cloud after registration transformation. Detailed Implementation
[0040] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.
[0041] See Figure 1 The diagram shown is a hardware architecture diagram of the preoperative and intraoperative liver point cloud data registration system 10 of the present invention. The system includes: a local hybrid feature extraction module 101, a global feature extraction module 102, a feature fusion module 103, an overlapping region mask prediction module 104, a transformation matrix prediction module 105, and a registration module 106.
[0042] in:
[0043] The local blending feature extraction module 101 is used to extract local blending features containing geometric structure information from preoperative point cloud data and intraoperative point cloud data, respectively. Specifically:
[0044] The local mixing feature f X It consists of three parts: the spatial coordinates of the point, the distance to neighboring points, and local geometric features. For a point X in the point cloud X... i Let its neighborhood be N(X) i If the local mixing characteristics are then... Represented as:
[0045]
[0046] Where f(·) represents a multilayer perceptron network used to extract local blending features of the point cloud, X j ∈N(X i ), This represents the overlapping region mask obtained in the previous iteration. This means converting neighborhood points into local neighbors after subtracting the centroid point:
[0047]
[0048] PPF(X i ,X j ) represents X j With X i The 4D point-pair spatial feature (PPF) between two spatial points is described using the distance between the two spatial points and the angle between their normal vectors.
[0049]
[0050] Where, n i and n j Representing point X i With X j The normal vector.
[0051] In the local blending feature extraction module 101, X is first... i The neighborhood N(X) i The features of all points within the range are transformed into... The 10-D feature vector is then passed through a series of convolutional layers, max pooling layers, and an output convolutional layer to obtain point X. i Final local blending features
[0052] The global feature extraction module 102 is used to fuse the extracted local mixed features to obtain the global features of the preoperative point cloud and the global features of the intraoperative point cloud. Specifically:
[0053] The global features of point cloud X are derived from its local blended features. In this embodiment, the local blended feature sets of each point can filter out some redundant information, thereby making the extracted global features more representative. The local blended features of all points in point cloud X in the i-th iteration are extracted. Then, the local mixed feature set is fed into a three-layer convolutional network, expanded to 1024-D, and then combined with the overlapping region mask obtained from the previous iteration. Multiply the results and pass them through a max-pooling layer to obtain the final global features.
[0054] The three-layer convolutional network has output dimensions of 96, 128, and 1024 for each layer; the local blending feature set is a set of local blending features. A set that consists of.
[0055] The feature fusion module 103 is used to fuse the local mixed features and global features of the preoperative point cloud and the global features of the intraoperative point cloud to obtain the fused features of the preoperative point cloud; similarly, it fuses the local mixed features and global features of the intraoperative point cloud and the global features of the preoperative point cloud to obtain the fused features of the intraoperative point cloud. Specifically:
[0056] After the local feature extraction module 101 and the global feature extraction module 102 obtain the local and global features of the two point clouds, the feature fusion module 103 contains three convolutional layers, which take the local feature of the preoperative point cloud X as input. Global features and global features of intraoperative point cloud Y Output fusion features of 512-dimensional preoperative point cloud Similarly, the fusion features of the intraoperative point cloud are obtained.
[0057] The overlapping region mask prediction module 104 is used to fuse the fusion features of the obtained preoperative point cloud and the fusion features of the intraoperative point cloud to obtain their respective overlapping region masks and decoding features.
[0058] Specifically:
[0059] In point cloud overlapping region mask prediction, the global features of point clouds X and Y... and and point-by-point features and The concatenation and interactive feedback in each iteration continuously improve the local mixed features. and The features are then adjusted and transformed, and then fused by the overlapping region mask prediction module 104 to obtain a new overlapping region mask. and regression features used to calculate the transformation matrix
[0060]
[0061]
[0062] Where g(·) represents the feature fusion module 103, f(·) represents the overlapping region mask prediction module 104, and the input fused features are... After four convolutional layers, the output is the predicted overlapping region mask. The decoded features of the point cloud are obtained by concatenating the outputs of the first three layers.
[0063] The transformation matrix prediction module 105 is used to multiply the overlapping region mask of the preoperative point cloud and the intraoperative point cloud with their respective local blending features, and link them with their respective decoding features to obtain the spatial transformation matrix of the preoperative point cloud and the intraoperative point cloud. Specifically:
[0064] Feature fusion module 103 obtains fused features obtained by fusing local and global features of the point cloud. The overlapping region mask prediction module 104 obtains layer-by-layer stitching features. The fused features and the layer-by-layer concatenation features are concatenated to obtain the final features required for spatial transformation prediction. The transformation matrix prediction module 105 contains five convolutional layers. The input features are fed into the transformation matrix prediction module 105 after a single max-pooling operation. After five convolutional layers, a 7-d feature vector is output, representing the spatial transformation of the current iteration, where the first four values represent the rotation quaternions q∈R. 4 The last three values represent the translation vector t∈R 3 The input feature refers to the feature formed by combining the fused feature and the layer-by-layer concatenation feature.
[0065]
[0066] in, and This represents the overlapping region mask predicted in the previous iteration, and p(·) represents the transformation matrix prediction module 105. After the iteration ends, the spatial transformations predicted in each round will be accumulated and calculated to obtain the final overall transformation between the preoperative point cloud and the intraoperative point cloud.
[0067] The registration module 106 is used to apply the calculated spatial transformation matrix to the preoperative point cloud data to obtain the registration result between the preoperative point cloud data and the intraoperative point cloud data. The preoperative point cloud data is the preoperative point cloud after registration transformation in the previous iteration; the registration result refers to registering the preoperative point cloud data to the intraoperative point cloud data using the transformation matrix.
[0068] Specifically:
[0069] Once the spatial transformation matrix T between the preoperative and intraoperative point clouds is determined, the transformation process can begin. Specifically, the virtual position of the preoperative point cloud within the intraoperative point cloud region is denoted as P. T The original location of the preoperative point cloud is denoted as P. I Then we have:
[0070] P T =T·P I (7)
[0071] This embodiment is executed four times, that is, iterated four times, to improve the registration accuracy, thereby registering the preoperative point cloud data to the intraoperative point cloud data. In other embodiments, iterations of 1, 2, or 3 times can also achieve the present application, and are also within the scope of protection of this application.
[0072] See Figure 2 The diagram shown is a flowchart of a preferred embodiment of the preoperative and intraoperative liver point cloud data registration method of the present invention.
[0073] Step S1: The local blending feature extraction module 101 extracts local blending features containing geometric structure information from the preoperative point cloud data and the intraoperative point cloud data, respectively. Specifically:
[0074] The local mixing feature f X It consists of three parts: the spatial coordinates of the point, the distance to neighboring points, and local geometric features. For a point X in the point cloud X... i Let its neighborhood be N(X) i If the local mixing characteristics are then... Represented as:
[0075]
[0076] Where f(·) represents a multilayer perceptron network used to extract local blending features of the point cloud, X j ∈N(X i ), This represents the overlapping region mask obtained in the previous iteration. This means converting neighborhood points into local neighbors after subtracting the centroid point:
[0077]
[0078] PPF(X i ,X j ) represents X j With X i The 4D point-pair spatial feature (PPF) between two spatial points is described using the distance between the two spatial points and the angle between their normal vectors.
[0079]
[0080] Where, n i and n j Representing point X i With X j The normal vector.
[0081] In the local blending feature extraction module 101, X is first... i The neighborhood N(X) i The features of all points within the range are transformed into... The 10-D feature vector is then passed through a series of convolutional layers, max pooling layers, and an output convolutional layer to obtain point X. i Final local blending features
[0082] Step S2: The global feature extraction module 102 fuses the extracted local mixed features to obtain the global features of the preoperative point cloud and the global features of the intraoperative point cloud. Specifically:
[0083] The global features of point cloud X are derived from its local blended features. In this embodiment, the local blended feature sets of each point can filter out some redundant information, thereby making the extracted global features more representative. The local blended features of all points in point cloud X in the i-th iteration are extracted. Then, the local mixed feature set is fed into a three-layer convolutional network, expanded to 1024-D, and then combined with the overlapping region mask obtained from the previous iteration. Multiply the results and pass them through a max-pooling layer to obtain the final global features.
[0084] The three-layer convolutional network has output dimensions of 96, 128, and 1024 for each layer; the local blending feature set is a set of local blending features. A set that consists of.
[0085] Step S3: The feature fusion module 103 fuses the local mixed features and global features of the preoperative point cloud and the global features of the intraoperative point cloud to obtain the fused features of the preoperative point cloud; similarly, it fuses the local mixed features and global features of the intraoperative point cloud and the global features of the preoperative point cloud to obtain the fused features of the intraoperative point cloud. Specifically:
[0086] After steps S1 and S2, the local and global features of the two point clouds are obtained. The feature fusion module 103 contains three convolutional layers, which take the local features of the preoperative point cloud X as input. Global features and global features of intraoperative point cloud Y Output fusion features of 512-dimensional preoperative point cloud Similarly, the fusion features of the intraoperative point cloud are obtained.
[0087] Step S4 involves fusing the fusion features of the preoperative point cloud and the intraoperative point cloud to obtain their respective overlapping region masks and decoding features. Specifically:
[0088] In point cloud overlapping region mask prediction, the global features of point clouds X and Y... and and point-by-point features and The concatenation and interactive feedback in each iteration continuously improve the local mixed features. and The features are then adjusted and transformed, and then fused by the overlapping region mask prediction module 104 to obtain a new overlapping region mask. and regression features used to calculate the transformation matrix
[0089]
[0090]
[0091] Where g(·) represents the feature fusion module 103 in step S3, and f(·) represents the overlapping region mask prediction module 104, with the input fused features... After four convolutional layers, the output is the predicted overlapping region mask. The decoded features of the point cloud are obtained by concatenating the outputs of the first three layers.
[0092] Step S5: The transformation matrix prediction module 105 multiplies the overlapping region mask of the preoperative point cloud and the intraoperative point cloud with their respective local blending features, and links them with their respective decoding features to obtain the spatial transformation matrix of the preoperative point cloud and the intraoperative point cloud. Specifically:
[0093] Step S3 yields the fused features obtained by fusing local and global features of the point cloud. Step S4 yields the layer-by-layer splicing features. The fused features and the layer-by-layer concatenation features are concatenated to obtain the final features required for spatial transformation prediction. The transformation matrix prediction module 105 contains five convolutional layers. The input features are fed into the transformation matrix prediction module 105 after a single max-pooling operation. After five convolutional layers, a 7-d feature vector is output, representing the spatial transformation of the current iteration, where the first four values represent the rotation quaternions q∈R. 4 The last three values represent the translation vector t∈R 3 The input feature refers to the feature formed by combining the fused feature and the layer-by-layer concatenation feature.
[0094]
[0095] in, and This represents the overlapping region mask predicted in the previous iteration, and p(·) represents the transformation matrix prediction module 105. After the iteration ends, the spatial transformations predicted in each round will be accumulated and calculated to obtain the final overall transformation between the preoperative point cloud and the intraoperative point cloud.
[0096] Step S6: Apply the calculated spatial transformation matrix to the preoperative point cloud data to obtain the registration result between the preoperative point cloud data and the intraoperative point cloud data. The preoperative point cloud data is the preoperative point cloud after registration transformation in the previous iteration; the registration result refers to registering the preoperative point cloud data to the intraoperative point cloud data using the transformation matrix.
[0097] Specifically:
[0098] Once the spatial transformation matrix T between the preoperative and intraoperative point clouds is determined, the transformation process can begin. The specific steps are as follows: Let P be the virtual position of the preoperative point cloud within the intraoperative point cloud region. T The original location of the preoperative point cloud is denoted as P. I Then we have:
[0099] P T =T·P I (7)
[0100] This embodiment is executed four times, that is, steps S1-S7 are iterated four times to improve registration accuracy, thereby registering the preoperative point cloud data to the intraoperative point cloud data. In other embodiments, iterations of 1, 2, or 3 times can also achieve the present application, and are also within the scope of protection of this application.
[0101] Please see Figure 3 This is a schematic diagram of a terminal structure according to an embodiment of this application. The terminal 50 includes a processor 51 and a memory 52 coupled to the processor 51.
[0102] The memory 52 stores program instructions for implementing the above-described preoperative and intraoperative liver point cloud data registration method.
[0103] The processor 51 is used to execute program instructions stored in the memory 52 to perform preoperative and intraoperative liver point cloud data registration.
[0104] The processor 51 can also be referred to as a CPU (Central Processing Unit). The processor 51 may be an integrated circuit chip with signal processing capabilities. The processor 51 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor.
[0105] Please see Figure 4This is a schematic diagram of the structure of the storage medium in an embodiment of this application. The storage medium in this embodiment stores a program file 61 capable of implementing all the above methods. This program file 61 can be stored in the storage medium in the form of a software product, including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, or terminal devices such as computers, servers, mobile phones, and tablets.
[0106] This application has been verified by experimental data of 3D laparoscopic liver point cloud data. The experiment shows that the application can avoid potential errors in the registration of preoperative liver point cloud data and intraoperative 3D laparoscopic liver surface point cloud data, improve the registration accuracy when the overlap of point cloud data is low, and the registration process is repeatable.
[0107] The test set used in this application includes 13 pairs of preoperative and intraoperative point cloud data of pig liver surface under different deformation states. The test set was taken from a 50 kg male white pig. Richard et al. used laparotomy to evenly distribute 15 1 cm long metal clips on the ventral surface of the liver and implanted 45 metal balls with a diameter of 2 mm inside the liver to calculate the registration TRE to evaluate the accuracy. After injecting standard CO2 pressure (12 mmHg) into the abdominal cavity, the four pig liver lobes were moved to different positions to obtain 13 different laparoscopic configurations. The laparoscopic surface data of each configuration was recorded in the video using a three-dimensional camera system. The intraoperative liver surface data was obtained by 3D surface reconstruction of the laparoscopic video of each configuration using the dense SfM method with photoscanpro (4) software. photoscanpro is a state-of-the-art surface reconstruction software. Preoperative liver surface point cloud data were obtained from computed tomography (CT) scans of each configuration. The experimental pigs were kept in a state of apnea during data acquisition. Preoperative and intraoperative liver point cloud data were already registered. In this experiment, 2048 points were randomly sampled from the preoperative and intraoperative point cloud data respectively as experimental point cloud data, and the registration accuracy was evaluated using the positional information of 15 metal clips on the ventral surface of the liver.
[0108] The main evaluation metrics used are the target registration error (TRE) for registration point error and the isotropic rotation error and translation error for registration transformation matrix.
[0109] Target registration error (TRE) represents the distance between corresponding points other than the reference point after registration.
[0110]
[0111] Where (x) gt ,y gt ,z gt (x) represents the coordinates of the ground truth point X in the point cloud. pred ,y pred ,z pred ) represents the coordinates of the point after registration transformation.
[0112] Isotropic rotation error and translation error represent the errors between the predicted rotation matrix and translation vector and the true value:
[0113] Error(t) = ||t gt -t pred ||2
[0114] Where {R gt ,t gt} and {R pred ,t pred} represent the true value and predicted value of the rotation and translation matrix, respectively.
[0115] In addition, the root mean square error (RMSE) and mean square error (MAE) of the rotation matrix and translation vector were calculated as reference indicators.
[0116] Table 1 compares the TRE scores of this application, reference [1], and reference [2] on the test set. The last column shows the average TRE error of the entire test set, and the best result is marked in bold. Reference [2] failed to register on this data pair and did not obtain the correct result, so its sixth column TRE error is marked as "--".
[0117] Table 1 Comparison of TRE indicators in this application, reference [1] and reference [2]
[0118]
[0119] Table 2 compares the errors of the three methods on the transformation matrix, where Error(R) and Error(t) represent isotropic rotation and translation errors, RMSE(R) and RMSE(t) represent the root mean square error (RMSE) of the rotation matrix and translation vector, and Error(R) and Error(t) represent the average error of the two methods.
[0120] Table 2 Comparison of transformation matrix errors in this application, reference [1] and reference [2]
[0121]
[0122] Table 3 compares the registration success rates of the three methods.
[0123] Table 3 compares the registration success rates of this application, reference [1], and reference [2].
[0124]
[0125] Figure 5 The registration visualization results of the three algorithms are shown in (a) of this application, (b) of reference [1], and (c) of reference [2].
[0126] This application utilizes a learned overlapping region mask to filter out non-overlapping regions, transforming partial-to-partial point cloud registration into registration of the same shape. Then, it registers the extracted overlapping region point clouds based on local blending features and global features. This application is better suited for registering liver point cloud data under 3D laparoscopy. Applying this application in 3D laparoscopic liver tumor resection allows for the registration of the liver surface reconstructed from preoperative CT images and intraoperative 3D laparoscopic images. This facilitates precise localization and resection of liver tumors, shortens surgical time, increases tumor resection rate, and improves surgical precision and safety.
[0127] Although the present invention has been described with reference to the present preferred embodiments, those skilled in the art should understand that the above preferred embodiments are only used to illustrate the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A preoperative and intraoperative liver point cloud data registration system, characterized in that, include: Local blending feature extraction module: used to extract local blending features containing geometric structure information from preoperative point cloud data and intraoperative point cloud data respectively; Global feature extraction module: used to fuse the extracted local mixed features to obtain the global features of the preoperative point cloud and the global features of the intraoperative point cloud; Feature fusion module: This module is used to fuse the local mixed features and global features of the preoperative point cloud and the global features of the intraoperative point cloud to obtain the fused features of the preoperative point cloud; similarly, it fuses the local mixed features and global features of the intraoperative point cloud and the global features of the preoperative point cloud to obtain the fused features of the intraoperative point cloud. The overlapping region mask prediction module is used to fuse the fusion features of the obtained preoperative point cloud and the fusion features of the intraoperative point cloud to obtain their respective overlapping region masks and decoding features. The transformation matrix prediction module is used to multiply the overlapping region mask of the preoperative point cloud and the intraoperative point cloud with their respective local blending features, and link them with their respective decoding features to obtain the spatial transformation matrix of the preoperative point cloud and the intraoperative point cloud. The registration module applies the calculated spatial transformation matrix to the preoperative point cloud data to obtain the registration result between the preoperative and intraoperative point cloud data; where: The aforementioned local mixing features It consists of three parts: the spatial coordinates of the point, the distance to neighboring points, and local geometric features; for point clouds Points in Let its neighborhood be Then its local mixing characteristics Represented as: in, This represents a multilayer perceptron network used to extract local blended features from a point cloud. , This represents the overlapping region mask obtained in the previous iteration. This means converting neighborhood points into local neighbors after subtracting the centroid point: express and The 4D point-pair spatial feature (PPF) between two spatial points is described using the distance between the two spatial points and the angle between their normal vectors. in, and Point and The normal vector.
2. The preoperative and intraoperative liver point cloud data registration system as described in claim 1, characterized in that: The local hybrid feature extraction module, the global feature extraction module, the feature fusion module, the overlapping region mask prediction module, the transformation matrix prediction module, and the registration module are executed sequentially once, which is one iteration, for a total of 4 iterations.
3. The preoperative and intraoperative liver point cloud data registration system as described in claim 2, characterized in that, The global feature extraction module is specifically used for: After extracting the first Mid-point cloud in round iteration Local blending features of all points within Then, the local mixed feature set is fed into a three-layer convolutional network, expanded to 1024-D, and then combined with the overlapping region mask obtained from the previous iteration. Multiply the results and pass them through a max pooling layer to obtain the final global features. .
4. The preoperative and intraoperative liver point cloud data registration system as described in claim 3, characterized in that, The feature fusion module contains three convolutional layers, taking the preoperative point cloud as input. Local mixing features Global features and intraoperative point cloud global features Output fusion features of 512-dimensional preoperative point cloud Similarly, the fusion features of the intraoperative point cloud are obtained. .
5. The preoperative and intraoperative liver point cloud data registration system as described in claim 4, characterized in that, The overlapping region mask prediction module is specifically used for: Point cloud and global features and and point-by-point features and The concatenation and interactive feedback in each iteration continuously improve the local mixed features. and The above features are then adjusted and transformed, and then fused to obtain a new overlapping region mask. , and regression features used to calculate the transformation matrix , ; in, Indicates the feature fusion module, This represents the overlapping region mask prediction module, with the input fused features. , After four convolutional layers, the output is the predicted overlapping region mask. The decoded features of the point cloud are obtained by concatenating the outputs of the first three layers. , .
6. The preoperative and intraoperative liver point cloud data registration system as described in claim 5, characterized in that: The transformation matrix prediction module contains 5 convolutional layers. The input features are fed into the transformation matrix prediction module after a single max pooling operation. After 5 convolutional layers, a 7-d feature vector is output, representing the spatial transformation of the current iteration, where the first 4 values represent the rotation quaternions. The last three values represent the translation vector. The input feature refers to the feature formed by combining the fused feature and the layer-by-layer concatenation feature; in, and This represents the overlapping region mask predicted in the previous iteration. This indicates that after the iteration ends, the spatial transformation obtained from each round of prediction in the transformation matrix prediction module will be accumulated and calculated to obtain the final overall transformation between the preoperative point cloud and the intraoperative point cloud.
7. The preoperative and intraoperative liver point cloud data registration system as described in claim 6, characterized in that, The registration module is specifically used for: The spatial transformation matrix between the preoperative point cloud and the intraoperative point cloud was determined. T Then the transformation process can begin, specifically: the virtual position of the preoperative point cloud in the intraoperative point cloud region is denoted as... P T The original location of the preoperative point cloud is recorded as P I Then we have: 。 8. A method for preoperative and intraoperative liver point cloud data registration using the preoperative and intraoperative liver point cloud data registration system of claim 1, characterized in that, The method includes the following steps: a. Extract local hybrid features containing geometric structure information from preoperative point cloud data and intraoperative point cloud data respectively; b. The extracted local mixed features are fused to obtain the global features of the preoperative point cloud and the global features of the intraoperative point cloud; c. The local blending features, global features of the preoperative point cloud, and global features of the intraoperative point cloud are fused to obtain the fused features of the preoperative point cloud; similarly, the local blending features, global features of the intraoperative point cloud, and global features of the preoperative point cloud are fused to obtain the fused features of the intraoperative point cloud. d. The fusion features of the preoperative point cloud and the fusion features of the intraoperative point cloud are fused to obtain their respective overlapping region masks and decoding features; e. Multiply the overlapping region mask of the preoperative point cloud and the intraoperative point cloud with their respective local blending features, and link them with their respective decoding features to obtain the spatial transformation matrix of the preoperative point cloud and the intraoperative point cloud. f. Apply the calculated spatial transformation matrix to the preoperative point cloud data to obtain the registration results between the preoperative point cloud data and the intraoperative point cloud data.
9. The method for preoperative and intraoperative liver point cloud data registration as described in claim 8, characterized in that, The method further includes iterating steps a through f four times.
10. A terminal, characterized in that, The terminal includes a processor and a memory coupled to the processor, wherein: The memory stores program instructions for implementing the preoperative and intraoperative liver point cloud data registration method according to any one of claims 8-9; The processor is used to execute the program instructions stored in the memory to achieve preoperative and intraoperative liver point cloud data registration.
11. A storage medium, characterized in that, The device stores processor-executable program instructions for performing the preoperative and intraoperative liver point cloud data registration method according to any one of claims 8-9.