A preoperative computed tomography image and intraoperative endoscope video registration method based on multi-anchor point fusion and related device

Abstract

This application provides a method and related apparatus for registering preoperative computed tomography (CT) images and intraoperative laparoscopic videos based on multi-anchor point fusion. The method includes: S1. acquiring preoperative CT image data and intraoperative laparoscopic video data; S2. extracting multiple types of anchor points from the two types of data, wherein the multiple types of anchor points include at least anatomical landmark anchor points, vascular branch anchor points, and tissue surface anchor points; S3. performing anchor point pairing based on the fusion features of the multiple types of anchor points; S4. solving the registration transformation matrix based on the paired anchor points; S5. evaluating the registration accuracy based on the residuals; S6. outputting the registration transformation matrix and a registration accuracy report. This application also provides related apparatus corresponding to the method, including devices, electronic devices, computer-readable storage media, and computer program products.

Description

Technical Field

[0001] This application relates to the field of medical image processing, and in particular to a method and related apparatus for registering preoperative computed tomography images and intraoperative laparoscopic videos based on multi-anchor point fusion. Background Technology

[0002] Precise registration of preoperative computed tomography (CT) images with intraoperative laparoscopic video is a key technology for achieving augmented reality surgical navigation, preoperative planning and intraoperative execution, and precise intraoperative localization. CT images provide three-dimensional information of deep anatomical structures, while laparoscopic video provides real-time intraoperative surface views. Precise registration of the two enables the surgeon to obtain information about deep structures within the intraoperative field of view.

[0003] Existing CT-endoscopic registration methods have the following problems: First, existing methods mainly perform registration based on a single type of anchor point (such as surface feature points or anatomical landmarks). The visibility and stability of a single anchor point vary greatly at different surgical stages, resulting in insufficient registration robustness. Second, existing methods do not introduce type consistency constraints and geometric consistency constraints for anchor point pairing, which easily leads to incorrect pairing. Third, the registration accuracy assessment of existing methods is insufficient, and no complete residual statistical indicators are provided, making it impossible for surgeons to accurately judge the registration quality. Fourth, the registration result output format of existing methods is not standardized, making it difficult to interface with downstream surgical navigation systems. Fifth, existing methods do not provide a confidence labeling mechanism based on accuracy thresholds. Summary of the Invention

[0004] To address the above technical issues, this application provides a method and related apparatus for registering preoperative computed tomography images and intraoperative laparoscopic videos based on multi-anchor point fusion.

[0005] Firstly, a method for registering preoperative computed tomography images and intraoperative laparoscopic videos based on multi-anchor point fusion is provided, comprising the following steps: S1. Acquire preoperative computed tomography (CT) image data and intraoperative laparoscopic video data, wherein the preoperative CT image data is referred to as preoperative data; S2. Extract multiple types of anchor points from the preoperative computed tomography image data and the intraoperative laparoscopic video data, respectively. The multiple types of anchor points include at least anatomical landmark anchor points, vascular branch anchor points, and tissue surface anchor points. The extraction of multiple types of anchor points is based on a key point detection algorithm. S3. Based on the fusion features of the multi-type anchor points, perform anchor point pairing to obtain a preoperative-intraoperative paired anchor point set. The fusion feature pairing is performed based on the similarity of the fusion features. S4. Solve the registration transformation matrix based on the preoperative-intraoperative paired anchor point set. The registration transformation matrix includes rotation, translation and scaling parameters. The solution is based on the least squares method. S5. Based on the registration transformation matrix and the residuals of the verification anchor points that were not involved in the solution, the registration accuracy is evaluated to obtain the registration accuracy evaluation index. The registration accuracy evaluation index includes the average residual distance, the maximum residual distance, and the standard deviation of the residual distance. The registration accuracy evaluation index is marked with a registration accuracy report based on a preset accuracy threshold. S6. Output the registration transformation matrix and registration accuracy report.

[0006] In any embodiment of this application, the multi-type anchor points include at least anatomical landmark anchor points, vascular branch anchor points, and tissue surface anchor points.

[0007] In any embodiment of this application, the anchor point pairing introduces type consistency constraints and RANSAC-based geometric consistency constraints.

[0008] In any embodiment of this application, the registration transformation matrix is ​​solved based on the following formula: , The solution is performed based on the least squares method or the LM iterative algorithm.

[0009] In any embodiment of this application, the registration accuracy report includes the average residual distance, the maximum residual distance, the standard deviation of the residual distance, and a confidence level.

[0010] In a second aspect, a device for registering preoperative computed tomography images and intraoperative laparoscopic videos based on multi-anchor point fusion is provided, comprising: a CT and laparoscopic data acquisition unit, a multi-anchor point extraction unit, an anchor point pairing unit, a transformation matrix solving unit, a registration accuracy evaluation unit, and a registration result output unit, which respectively execute steps S1 to S6 of the method described in the first aspect.

[0011] Thirdly, an electronic device is provided, including a processor and a storage unit, the storage unit being used to store computer program code, the code containing computer instructions, wherein when the processor executes these instructions, the electronic device performs the method described in the first aspect.

[0012] Fourthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in the first aspect.

[0013] Fifthly, a computer program product is provided, the computer program product comprising a computer program or instructions that, when the computer program or instructions are run on a computer, cause the computer to perform the method described in the first aspect.

[0014] It should be understood that the above general descriptions and subsequent specific descriptions are for illustrative and explanatory purposes only and do not impose any limitations on this application.

[0015] In this application, compared with the prior art, the technical problem to be solved by this application is: to achieve a highly robust registration based on multi-type anchor point fusion in the registration of preoperative CT images and intraoperative laparoscopic videos, to address the problem of insufficient stability of single-type anchor points at different surgical stages, to introduce type consistency constraints and geometric consistency constraints to avoid incorrect pairing, to provide a complete registration accuracy evaluation index, and to provide a confidence label based on accuracy threshold. The technical means employed in this application and their non-obvious aspects compared to existing technologies include: First, this application addresses the problem of insufficient stability of a single anchor point based on a multi-type anchor point (anatomical landmarks + vascular branches + tissue surfaces) fusion mechanism, a design not seen in existing methods using a single type of anchor point; Second, this application introduces anchor point type consistency constraints to avoid incorrect pairing between anchor points of different types, a design not seen in existing methods without type constraints; Third, this application performs geometric consistency constraints to eliminate outlier pairings based on the RANSAC algorithm, a design not seen in existing methods relying solely on feature similarity; Fourth, this application provides complete residual statistical indicators (mean residual, maximum residual, residual standard deviation), a design not seen in existing methods that only output a single precision indicator; Fifth, this application labels registration confidence based on a preset precision threshold, a design not seen in existing methods that only output registration results without confidence feedback. The related devices described in this application include devices, electronic devices, computer-readable storage media, and computer program products. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be explained below.

[0017] Figure 1 This is a flowchart illustrating a method for registering preoperative computed tomography images and intraoperative laparoscopic videos based on multi-anchor point fusion, as provided in an embodiment of this application.

[0018] Figure 2 This is a schematic diagram of a device for registering preoperative computed tomography images and intraoperative laparoscopic videos based on multi-anchor point fusion, provided in an embodiment of this application.

[0019] Figure 3 This is a schematic diagram of the hardware architecture of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] To enable those skilled in the art to more fully understand the technical solution of this application, the technical solution of this application will be explained in detail and clearly with reference to the accompanying drawings.

[0021] Please see Figure 1 , Figure 1 This is a flowchart illustrating a method for registering preoperative computed tomography images and intraoperative laparoscopic videos based on multi-anchor point fusion, as provided in an embodiment of this application. The method includes steps S1 to S6.

[0022] S1. CT and Laparoscopic Data Acquisition: Acquire preoperative computed tomography (CT) image data and intraoperative laparoscopic video data.

[0023] In this embodiment, the preoperative computed tomography (CT) image data can be single-phase CT or multi-phase enhanced CT, and the image data has been reconstructed in three dimensions. The intraoperative laparoscopic video data is acquired in real time by an intracavitary imaging device during the surgical procedure.

[0024] S2. Multiple anchor point extraction: Extract multiple types of anchor points from the preoperative data and intraoperative data respectively.

[0025] In this embodiment, the anatomical landmark anchor points include at least one of the following: hepatic hilum, gallbladder bed, root of mesentery, and ileocecal junction. The vascular branch anchor points include at least one of the following: superior mesenteric artery bifurcation point, inferior mesenteric artery bifurcation point, and ileocolic artery bifurcation point. The tissue surface anchor points include at least high-response surface feature points extracted based on a keypoint detection algorithm.

[0026] The extraction of anatomical landmark anchors and vascular branch anchors is performed using a deep learning segmentation and localization network. The extraction of tissue surface anchors is performed using a keypoint detection algorithm, which includes, but is not limited to, SIFT, SURF, ORB, or a deep learning-based keypoint detection network.

[0027] S3, Anchor point pairing: Perform anchor point pairing based on the fusion features of the multiple types of anchor points.

[0028] In this embodiment, the anchor point pairing is performed based on the following rules: for each preoperative anchor point, the anchor point with the highest fusion feature similarity in the intraoperative anchor point set is selected as a pairing candidate; pairings with mismatched types are filtered out based on anchor point type consistency constraints; and outlier pairings with inconsistent geometric transformations are eliminated based on the RANSAC algorithm. The fusion feature similarity is calculated based on the weighted distance of anchor point position, morphology, and contextual features.

[0029] S4. Solving the transformation matrix: Solve the registration transformation matrix based on the preoperative-intraoperative paired anchor point set.

[0030] In this embodiment, the registration transformation matrix is ​​solved based on the following formula: , Where T* is the optimal registration transformation matrix, T is the transformation matrix to be solved, p_CT_i are the preoperative anchor point coordinates, p_endo_i are the corresponding intraoperative anchor point coordinates, and Σ is the summation over all paired anchor points. The registration transformation matrix is ​​parameterized based on at least one of rigid transformation, similarity transformation, or affine transformation. The solution is performed using the least squares method or the LM iterative algorithm.

[0031] S5. Registration accuracy assessment: The registration accuracy is assessed based on the registration transformation matrix and the residuals of the verification anchor points that were not involved in the solution.

[0032] In this embodiment, the registration accuracy assessment is calculated based on the following indicators: average residual distance, maximum residual distance, and standard deviation of residual distance.

[0033] S6. Registration Result Output: Output the registration transformation matrix and registration accuracy report.

[0034] In this embodiment, the registration accuracy report includes at least the registration transformation matrix, the average residual distance, the maximum residual distance, the standard deviation of the residual distance, and the number of paired anchor points involved in the solution. When the average residual distance is greater than a preset accuracy threshold (ranging from 3 mm to 8 mm), the registration accuracy report is marked as unreliable.

[0035] Application Example 1: In laparoscopic liver resection surgery, the method registers the three-dimensional model of liver vessels constructed by preoperative CT with the intraoperative laparoscopic video. Based on the fusion of multiple types of anchor points, including the hepatic hilum (anatomical landmark anchor point), hepatic artery bifurcation (vascular branch anchor point), and liver surface feature points (tissue surface anchor points), the average residual distance of registration reaches 3.5 mm, assisting the surgeon in accurately identifying the course of deep hepatic veins and avoiding vascular damage.

[0036] Application Example 2: In laparoscopic radical colectomy, the method registers the mesenteric vascular model constructed by preoperative CTA with the intraoperative laparoscopic video to help the surgeon identify the branching structure of the superior mesenteric artery and the distribution of lymph nodes.

[0037] Please see Figure 2 , Figure 2 This application provides a schematic diagram of a preoperative computed tomography (CT) image and intraoperative laparoscopic video registration device based on multi-anchor point fusion. The device 1 includes: The CT and laparoscopic data acquisition unit 11 is used to acquire preoperative computed tomography image data and intraoperative laparoscopic video data.

[0038] The multi-anchor extraction unit 12 is used to extract multiple types of anchors from the two types of data respectively.

[0039] Anchor point pairing unit 13 is used to perform anchor point pairing based on the fusion features of multiple types of anchor points.

[0040] Transformation matrix solving unit 14 is used to solve the registration transformation matrix based on paired anchor points.

[0041] Registration accuracy evaluation unit 15 is used to evaluate registration accuracy based on residuals.

[0042] The registration result output unit 16 is used to output the registration transformation matrix and the registration accuracy report.

[0043] Please see Figure 3 , Figure 3 This is a schematic diagram of the hardware architecture of an electronic device provided in an embodiment of this application. The electronic device 2 includes a processor 21, a memory 22, an input device 23, and an output device 24. The processor 21, the memory 22, the input device 23, and the output device 24 are connected in communication via a bus.

[0044] The method described in this application can be implemented based on hardware, software, or a combination of hardware and software.

Claims

1. A method for registering preoperative computed tomography images and intraoperative laparoscopic videos based on multi-anchor point fusion, characterized in that... It includes the following steps: S1. Acquire preoperative computed tomography (CT) image data and intraoperative laparoscopic video data, wherein the preoperative CT image data is referred to as preoperative data; S2. Extract multiple types of anchor points from the preoperative computed tomography image data and the intraoperative laparoscopic video data, respectively. The multiple types of anchor points include at least anatomical landmark anchor points, vascular branch anchor points, and tissue surface anchor points. The extraction of multiple types of anchor points is based on a key point detection algorithm. S3. Based on the fusion features of the multi-type anchor points, perform anchor point pairing to obtain a preoperative-intraoperative paired anchor point set. The fusion feature pairing is performed based on the similarity of the fusion features. S4. Solve the registration transformation matrix based on the preoperative-intraoperative paired anchor point set. The registration transformation matrix includes rotation, translation, and scaling parameters. The solution is based on the least squares method. S5. Based on the registration transformation matrix and the residuals of the verification anchor points not involved in the solution, assess the registration accuracy to obtain registration accuracy evaluation indicators. The registration accuracy evaluation indicators include the average residual distance, the maximum residual distance, and the standard deviation of the residual distance. Based on a preset accuracy threshold, mark the registration accuracy evaluation indicators with registration accuracy reports. S6. Output the registration transformation matrix and registration accuracy report.

2. The method according to claim 1, characterized in that... The anatomical landmark anchor points include at least one of the following: hepatic hilum, gallbladder bed, mesenteric root, and ileocecal junction; the vascular branch anchor points include at least one of the following: superior mesenteric artery bifurcation point, inferior mesenteric artery bifurcation point, and ileocolic artery bifurcation point; and the tissue surface anchor points include at least high-response surface feature points extracted based on a key point detection algorithm.

3. The method according to claim 1, characterized in that... The extraction of anatomical landmark anchors and vascular branch anchors is performed based on a deep learning segmentation and localization network; the extraction of tissue surface anchors is performed based on a key point detection algorithm, which includes SIFT, SURF, ORB, or a deep learning-based key point detection network.

4. The method according to claim 1, characterized in that... The anchor point pairing is performed based on the following rules: for each preoperative anchor point, the anchor point with the highest fusion feature similarity is found in the intraoperative anchor point set as a pairing candidate; pairings with mismatched types are filtered out based on anchor point type consistency constraints; outlier pairings with inconsistent geometric transformations are eliminated based on the RANSAC algorithm; the fusion feature similarity is calculated based on the weighted distance of anchor point position, morphology and context features.

5. The method according to claim 1, characterized in that... The registration transformation matrix is ​​solved based on the following formula: ， Where T* is the optimal registration transformation matrix, T is the transformation matrix to be solved, p_CT_i is the preoperative anchor point coordinate, p_endo_i is the corresponding intraoperative anchor point coordinate, and Σ is the summation over all paired anchor points; the registration transformation matrix is ​​parameterized based on at least one of rigid transformation, similarity transformation or affine transformation; the solution is performed based on the least squares method or the LM iterative algorithm.

6. The method according to claim 1, characterized in that... The registration accuracy assessment is calculated based on the following indicators: average residual distance, maximum residual distance, and standard deviation of residual distance; the registration accuracy report includes at least the registration transformation matrix, the average residual distance, the maximum residual distance, the standard deviation of residual distance, and the number of paired anchor points involved in the solution; when the average residual distance is greater than the preset accuracy threshold, the registration accuracy report is marked as unreliable; the preset accuracy threshold ranges from 3 mm to 8 mm; the registration transformation matrix and the registration accuracy report are also used for at least one of the following downstream applications: (1) augmented reality preoperative model overlay; (2) surgical robot coordinate system calibration; (3) intraoperative navigation path planning; (4) surgical efficacy evaluation.

7. A device for registering preoperative computed tomography images and intraoperative laparoscopic videos based on multi-anchor point fusion, characterized in that... ,include: The CT and endoscopic data acquisition unit is used to perform step S1 as described in claim 1. A multi-anchor point extraction unit is used to perform step S2 as described in claim 1; An anchor point pairing unit, configured to perform step S3 as described in claim 1; A transformation matrix solving unit is used to perform step S4 as described in claim 1; A registration accuracy evaluation unit is used to perform step S5 as described in claim 1; The registration result output unit is used to perform step S6 as described in claim 1.

8. An electronic device, characterized in that... The electronic device includes: a processor and a storage unit for storing computer program code containing computer instructions, wherein when the processor executes these instructions, the electronic device performs the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that... The computer-readable storage medium stores a computer program, the computer program containing program instructions that, when executed by a processor, cause the processor to perform the method described in any one of claims 1 to 6.

10. A computer program product, characterized in that... The computer program product comprises a computer program or instructions that, when executed on a computer, cause the computer to perform the method described in any one of claims 1 to 6.

Images