Intraoperative pulmonary nodule positioning method and system based on fusion of CT image and endoscope video

Abstract

The invention discloses an intraoperative pulmonary nodule positioning method and system based on fusion of a CT image and an endoscope video. The method comprises the following steps: firstly, constructing a three-dimensional lung model containing a pulmonary nodule position based on preoperative high-resolution CT data; generating a real-time lung surface model by using an endoscope video stream through a three-dimensional reconstruction technology during the operation; a feature point extraction and matching algorithm is adopted, and a non-rigid registration technology is combined to realize accurate registration of the preoperative model and the intraoperative dynamic model; projection display of pulmonary nodules in a real-time endoscope visual field is established through space coordinate mapping, and a dynamic compensation algorithm is developed to solve the problem of tissue deformation caused by respiratory movement and instrument operation. Meanwhile, the virtual nodule position is overlaid to an endoscope video picture through the augmented reality technology, and visual intraoperative navigation is provided for surgeons. The method effectively solves the technical bottleneck that a traditional method depends on a metal marker and cannot adapt to intraoperative tissue deformation and the like, and has clinical practical value.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing and computer-assisted surgery, and specifically to a method and system for locating lung nodules during surgery based on the fusion of CT images and laparoscopic videos. Background Art

[0002] With the development of minimally invasive thoracic surgery technology, thoracoscopic surgery has become the mainstream means of diagnosis and treatment of lung nodules, and the precise positioning of lung nodules during surgery is the key prerequisite for the success of the operation. In current clinical practice, the positioning of lung nodules mainly adopts the method of preoperative needle puncture positioning combined with the doctor's experience and judgment during surgery. This method not only causes trauma to the patient, but also has the risk of complications such as pneumothorax and bleeding. In addition, the intraoperative positioning process is highly dependent on the surgeon's operating experience, and the positioning accuracy is restricted by factors such as the location and depth of the nodule, resulting in prolonged operation time. According to statistics, the average time for intraoperative secondary positioning due to preoperative positioning deviation is 10-30 minutes, which seriously affects the efficiency and safety of the operation.

[0003] Currently, CT-guided puncture localization remains a common clinical procedure, but this technology has significant limitations: the puncture process causes pain and discomfort to the patient, and the procedure itself is invasive. Furthermore, positioning errors are inevitable due to factors such as respiratory movement and subtle nodule displacement. Even after confirming the positioning needle's position during surgery, the surgeon still needs to rely on experience and repeated exploration to determine the actual location of the nodule. This process significantly increases the duration and potential risks of the operation, adversely affecting the patient's prognosis. Summary of the Invention

[0004] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of this application to avoid blurring the purpose of this section, the abstract and the title of the invention, and such simplifications or omissions should not be used to limit the scope of the present invention.

[0005] Therefore, the purpose of the present invention is to provide a method and system for locating lung nodules during surgery based on the fusion of CT images and laparoscopic videos, so as to solve the problems raised in the above background technology.

[0006] To solve the above technical problems, according to one aspect of the present invention, the present invention provides the following technical solutions:

[0007] The intraoperative lung nodule localization method based on CT image and laparoscopic video fusion has the following steps:

[0008] S1. Based on voxel data obtained from a preoperative CT scan, segment the voxel data by setting a HU value threshold, construct a three-dimensional lung surface model, and project the lung nodules onto the three-dimensional lung surface model;

[0009] S2. Divide the intraoperative laparoscopic video into a series of image frames, extract key image frames with significant feature changes, and establish a three-dimensional model of the local lung surface during the operation by matching feature points;

[0010] S3, matching the preoperative lung surface three-dimensional model with the intraoperative local lung surface three-dimensional model;

[0011] S4. Real-time display of the distance and direction between the center of the laparoscope and the location of the lung nodule.

[0012] As a preferred embodiment of the intraoperative lung nodule localization method based on CT image and laparoscopic video fusion described in the present invention, in step S1, a three-dimensional model of the lung surface is constructed using voxel data from a preoperative CT scan, and the lung nodules are projected onto the constructed three-dimensional lung surface model. The specific steps are as follows:

[0013] The voxel data obtained by CT scanning is subjected to threshold segmentation based on the Hu value characteristics of lung tissue, as shown in formula (1):

[0014]

[0015] Among them, B(x,y,z) represents the binary voxel data, h(x,y,z) represents the Hu value of the CT scan voxel data, τ low and τ high represents the minimum and maximum values ​​of the Hu threshold of lung tissue, and x, y, and z represent the three-dimensional coordinate data of the CT scan voxel data;

[0016] For the voxel data B(x, y, z) with a Hu value of 1, extract its outer surface triangular mesh, as shown in formula (2):

[0017] M={Δ j} (2)

[0018] Where M represents the lung surface triangulated network composed of binary voxel data, Δ j represents the j-th triangle;

[0019] The lung surface triangulation network is smoothed as shown in formula (3):

[0020] M'=S(M) (3)

[0021] Where M' represents the lung surface after smoothing, and S represents the smoothing of the lung surface triangulation network;

[0022] The lung nodules are projected onto the smoothed lung surface model M', as shown in formula (4):

[0023]

[0024] wherein p k represents the projection position of the kth lung nodule on the lung surface model, p represents the position of a point on the lung surface model, n k represents the position of the kth lung nodule, and M' represents the lung surface after smoothing processing.

[0025] The projection position p k of the lung nodule is labeled to obtain a lung surface three-dimensional model with lung nodule labeling.

[0026] As a preferred scheme of the intraoperative lung nodule positioning method based on CT image and endoscope video fusion according to the present application, in step S2, the intraoperative endoscope video is divided into a series of image frames, key image frames with significant feature changes are extracted, and the intraoperative local lung surface three-dimensional model is established through feature point matching, and the specific steps are as follows:

[0027] The gray level histogram similarity between the endoscope video image frames is calculated, as shown in formula (5):

[0028]

[0029] wherein S(I i ,I j ) represents the similarity between the image I i and I j , I i and I j represent the i th image frame and the j th image frame in the endoscope video, h i (k) and h j (k) represent the frequency of the gray level histogram of the image I i and I j at the gray level k, and k represents the gray value.

[0030] By comparing the similarity, a key image frame set with significant feature changes is extracted, K={I k1 ,I k2 ,...,I kn}, the number of key frames is n, for the n key frames, the feature points in each key frame are extracted, the three-dimensional coordinates of the extracted feature points are calculated by using the triangulation method, and the lung surface sparse point cloud is constructed.

[0031] As a preferred scheme of the intraoperative lung nodule positioning method based on CT image and endoscope video fusion according to the present application, in step S3, the matching of the preoperative lung surface three-dimensional model and the intraoperative local lung surface three-dimensional model, and the specific steps are as follows:

[0032] The transformation function of the preoperative lung surface three-dimensional model and the intraoperative local lung surface model is solved, as shown in formula (6):

[0033]

[0034] wherein, T * represents the optimal transformation function of the preoperative lung surface three-dimensional model and the intraoperative local lung surface three-dimensional model, T represents all possible transformations of the preoperative lung surface three-dimensional model and the intraoperative local lung surface three-dimensional model, V pre represents the preoperatively established lung surface three-dimensional model, V tra represents the intraoperatively established local lung surface three-dimensional model, v represents the key feature points in the preoperatively established lung surface three-dimensional model, and u represents the key feature points in the preoperatively established lung surface three-dimensional model.

[0035] As a preferred scheme of the intraoperative lung nodule positioning method based on CT image and endoscope video fusion according to the present application, in step S4, the step of displaying the distance and direction of the endoscope center from the lung nodule position in real time is as follows: the preoperative lung surface three-dimensional model and the intraoperative local lung surface three-dimensional model are displayed on the display unit in different colors and transparencies, and the distance and direction of the endoscope center from the lung nodule position are displayed in real time with the center of the endoscope probe as a reference.

[0036] An intraoperative lung nodule positioning system based on CT image and endoscope video fusion, comprising:

[0037] A lung surface three-dimensional model construction unit, which constructs a lung surface three-dimensional model by setting a HU value threshold to segment the voxel data obtained based on preoperative CT scanning, and projects a lung nodule onto the lung surface three-dimensional model;

[0038] An intraoperative local lung surface three-dimensional model construction unit, which divides the intraoperative endoscope video into a series of image frames, extracts key image frames with significant feature changes, and establishes an intraoperative local lung surface three-dimensional model through feature point matching;

[0039] A matching unit, which matches the preoperative lung surface three-dimensional model and the intraoperative local lung surface three-dimensional model;

[0040] A display unit, which displays the distance and direction of the endoscope center from the lung nodule position in real time.

[0041] Compared with existing technologies, the present invention has the following beneficial effects: Based on a positioning method for multimodal data fusion, the present invention achieves rapid positioning of lung nodules by fusing and matching preoperative CT scan data with real-time intraoperative laparoscopic video. The advantage of this technology is that it does not require additional puncture operations, completely avoiding the trauma caused to patients by traditional positioning methods; through real-time intraoperative positioning and navigation, it effectively shortens the surgeon's intraoperative exploration time for the location of the nodule, improving surgical efficiency. Compared with traditional methods, this solution effectively reduces surgical risks while ensuring non-invasive safety, providing more reliable technical support for minimally invasive treatment of lung nodules, and is expected to become a new technology for precise positioning in thoracoscopic surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and detailed embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be derived from these drawings without inventive effort. Among them:

[0043] Figure 1 This is a flow chart of the intraoperative lung nodule localization method based on the fusion of CT images and laparoscopic video of the present invention. DETAILED DESCRIPTION

[0044] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0045] The present invention provides a method and system for intraoperative lung nodule localization based on the fusion of CT images and laparoscopic videos, which can achieve rapid localization of lung nodules without the need for additional puncture operations, completely avoiding the trauma caused to patients by traditional positioning methods.

[0046] Example 1

[0047] like Figure 1 As shown in FIG, a real-time intraoperative pulmonary nodule localization method based on CT image and laparoscopic video fusion is performed as follows:

[0048] Step 1: Before surgery, the voxel data obtained from the patient's CT scan is threshold segmented based on the Hu value characteristics of the lung tissue, as shown in formula (1):

[0049]

[0050] Among them, B(x,y,z) represents the binary voxel data, h(x,y,z) represents the Hu value of the CT scan voxel data, τ low and τ highrepresents the minimum and maximum values ​​of the lung tissue Hu threshold, and x, y, and z represent the three-dimensional coordinate data of the CT scan voxel data.

[0051] For the voxel data B(x,y,z) with Hu value 1, extract its outer surface triangular mesh, as shown in formula (2):

[0052] M={Δ j} (2)

[0053] Where M represents the lung surface triangulated network composed of binary voxel data, Δ j Represents the jth triangle. It should be noted that the pipeline structures such as bronchi and blood vessels in the lungs are ignored.

[0054] The lung surface triangulation network is smoothed as shown in formula (3):

[0055] M'=S(M) (3)

[0056] Wherein, M' represents the lung surface after smoothing, and S represents the smoothing of the lung surface triangulation network.

[0057] The lung nodules are projected onto the smoothed lung surface model M', as shown in formula (4):

[0058]

[0059] Among them, p k represents the projection position of the kth lung nodule on the lung surface model, p represents the position of the point on the lung surface model, n k Represents the position of the kth lung nodule, M' represents the lung surface after smoothing. k After labeling, a three-dimensional model of the lung surface with lung nodules labeled can be obtained.

[0060] Step 2: During the operation, calculate the grayscale histogram similarity between the laparoscopic video image frames, as shown in formula (5):

[0061]

[0062] Among them, S(I i ,I j ) represents image I i and I j The similarity between i and I j represent the i-th image frame and the j-th image frame in the laparoscope video, respectively, h i (k) and h j (k) represents image I i and I jThe frequency of the grayscale histogram at grayscale level k, where k represents the grayscale value.

[0063] By comparing the similarity, a set of key image frames with significant feature changes is extracted, K = {I k1 ,I k2 ,...,I kn The number of key frames is n. For each of these n key frames, feature points are extracted, and the three-dimensional coordinates of the extracted feature points are calculated using triangulation, and a sparse point cloud of the lung surface is constructed.

[0064] Step 3: Solve the conversion function between the preoperative lung surface three-dimensional model and the intraoperative local lung surface model, as shown in formula (6):

[0065]

[0066] Among them, T * represents the optimal transformation function between the preoperative lung surface 3D model and the intraoperative local lung surface 3D model, T represents all possible transformations between the preoperative lung surface 3D model and the intraoperative local lung surface 3D model, V pre represents the three-dimensional model of the lung surface established before surgery, V tra represents the local lung surface 3D model created intraoperatively, v represents the key feature points in the preoperative 3D lung surface model, and u represents the key feature points in the preoperative 3D lung surface model. This formula indicates that the transformation T is sought to minimize the sum of squared distances between the vertices of the transformed preoperative model and the vertices of the intraoperative model, thereby achieving accurate matching of the two models. It is important to note that during the matching process, compensation for errors caused by respiratory motion and tissue shape must be considered.

[0067] Step 4: Display the preoperative lung surface three-dimensional model and the intraoperative local lung surface three-dimensional model on the display unit with different colors and transparencies. With the center of the laparoscope probe as a reference, display the distance and direction from the center of the laparoscope to the location of the lung nodule in real time.

[0068] Example 2

[0069] The present invention also provides an intraoperative lung nodule positioning system based on the fusion of CT images and laparoscopic videos to implement the steps of the intraoperative lung nodule positioning method based on the fusion of CT images and laparoscopic videos in Example 1. The system includes a lung surface three-dimensional model construction unit, an intraoperative local lung surface three-dimensional model construction unit, a matching unit and a display unit.

[0070] The lung surface three-dimensional model construction unit segments the voxel data obtained from the preoperative CT scan by setting a HU value threshold, constructs a lung surface three-dimensional model, and projects the lung nodules onto the lung surface three-dimensional model;

[0071] The intraoperative local lung surface 3D model construction unit divides the intraoperative laparoscopic video into a series of image frames, extracts key image frames with significant feature changes, and builds an intraoperative local lung surface 3D model by matching feature points.

[0072] The matching unit matches the preoperative lung surface three-dimensional model with the intraoperative local lung surface three-dimensional model;

[0073] The display unit displays the distance and direction between the center of the laparoscope and the location of the lung nodule in real time.

[0074] Although the present invention has been described above with reference to embodiments, various modifications may be made thereto and equivalent components may be substituted without departing from the scope of the present invention. In particular, as long as there are no structural conflicts, the various features of the embodiments disclosed herein may be combined with each other in any manner, and the omission of an exhaustive description of such combinations in this specification is solely for the sake of space and resource conservation. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A method for locating pulmonary nodules during surgery based on CT image and laparoscopic video fusion, characterized in that: Here are the steps: S1. Based on voxel data obtained from a preoperative CT scan, segment the voxel data by setting a HU value threshold, construct a three-dimensional lung surface model, and project the lung nodules onto the three-dimensional lung surface model; S2. Divide the intraoperative laparoscopic video into a series of image frames, extract key image frames with significant feature changes, and establish a three-dimensional model of the local lung surface during the operation by matching feature points; S3, matching the preoperative lung surface three-dimensional model with the intraoperative local lung surface three-dimensional model; S4. Real-time display of the distance and direction between the center of the laparoscope and the location of the lung nodule.

2. The intraoperative lung nodule localization method based on CT image and laparoscopic video fusion according to claim 1, characterized in that: In step S1, a three-dimensional model of the lung surface is constructed using voxel data from the preoperative CT scan, and lung nodules are projected onto the constructed three-dimensional model of the lung surface. The specific steps are as follows: The voxel data obtained by CT scanning is subjected to threshold segmentation based on the Hu value characteristics of lung tissue, as shown in formula (1): Among them, B(x,y,z) represents the binary voxel data, h(x,y,z) represents the Hu value of the CT scan voxel data, τ low and τ high represents the minimum and maximum values ​​of the Hu threshold of lung tissue, and x, y, and z represent the three-dimensional coordinate data of the CT scan voxel data; For the voxel data B(x,y,z) with Hu value 1, extract its outer surface triangular mesh, as shown in formula (2): M={Δ j } (2) Where M represents the lung surface triangulated network composed of binary voxel data, Δ j represents the j-th triangle; The lung surface triangulation network is smoothed as shown in formula (3): M'=S(M) (3) Where M' represents the lung surface after smoothing, and S represents the smoothing of the lung surface triangulation network; The lung nodules are projected onto the smoothed lung surface model M', as shown in formula (4): Among them, p k represents the projection position of the kth lung nodule on the lung surface model, p represents the position of the point on the lung surface model, n k represents the location of the kth lung nodule, and M' represents the lung surface after smoothing; Projection position of lung nodules p k The three-dimensional model of the lung surface with lung nodule annotations is obtained.

3. The intraoperative lung nodule localization method based on CT image and laparoscopic video fusion according to claim 1, characterized in that: In step S2, the intraoperative laparoscopic video is divided into a series of image frames, key image frames with significant feature changes are extracted, and a three-dimensional model of the local lung surface during the operation is established by matching feature points. The specific steps are as follows: Calculate the grayscale histogram similarity between laparoscope video image frames, as shown in formula (5): Among them, S(I i ,I j ) represents image I i and I j The similarity between i and I j represent the i-th image frame and the j-th image frame in the laparoscope video, respectively, h i (k) and h j (k) represents image I i and I j The frequency of the grayscale histogram at grayscale level k, where k represents the grayscale value; By comparing the similarity, a set of key image frames with significant feature changes is extracted, K = {I k1 ,I k2 ,...,I kn }, the number of key frames is n. For n key frames, the feature points in each key frame are extracted, the three-dimensional coordinates of the extracted feature points are calculated using triangulation, and a sparse point cloud of the lung surface is constructed.

4. The intraoperative lung nodule localization method based on CT image and laparoscopic video fusion according to claim 1, characterized in that: In step S3, the preoperative lung surface 3D model is matched with the intraoperative local lung surface 3D model. The specific steps are as follows: The conversion function between the preoperative lung surface three-dimensional model and the intraoperative local lung surface model is solved as shown in formula (6): Among them, T * represents the optimal transformation function between the preoperative lung surface 3D model and the intraoperative local lung surface 3D model, T represents all possible transformations between the preoperative lung surface 3D model and the intraoperative local lung surface 3D model, V pre represents the three-dimensional model of the lung surface established before surgery, V tra represents the local lung surface three-dimensional model established during the operation, v represents the key feature points in the lung surface three-dimensional model established before the operation, and u represents the key feature points in the lung surface three-dimensional model established before the operation.

5. The intraoperative lung nodule localization method based on CT image and laparoscopic video fusion according to claim 1, characterized in that: In step S4, the steps for displaying the distance and direction of the laparoscope center from the lung nodule position in real time are as follows: display the preoperative lung surface three-dimensional model and the intraoperative local lung surface three-dimensional model on the display unit with different colors and transparencies, and use the center of the laparoscope probe as a reference to display the distance and direction of the laparoscope center from the lung nodule position in real time.

6. A system for implementing the intraoperative lung nodule localization method based on CT image and laparoscopic video fusion according to any one of claims 1 to 5, characterized in that: include: A lung surface three-dimensional model construction unit, which segments the voxel data obtained from the preoperative CT scan by setting a HU value threshold, constructs a lung surface three-dimensional model, and projects the lung nodules onto the lung surface three-dimensional model; The intraoperative local lung surface 3D model construction unit divides the intraoperative laparoscopic video into a series of image frames, extracts key image frames with significant feature changes, and builds a 3D model of the intraoperative local lung surface by matching feature points; A matching unit matches the preoperative lung surface 3D model with the intraoperative local lung surface 3D model; The display unit displays the distance and direction between the center of the laparoscope and the location of the lung nodule in real time.

Images