A lung tumor position and shape real-time estimation method, system and storage medium

Abstract

This invention discloses a method, system, and storage medium for real-time estimation of lung tumor location and morphology, comprising: S1: acquiring preoperative end-expiratory and end-inspiratory CT images of the lungs, segmenting and reconstructing the skin and tumor surfaces respectively; S2: registering the end-expiratory skin and tumor surfaces to the end-inspiratory skin and tumor surfaces to obtain the full-process displacement vector fields of the skin and tumor surfaces; S3: during surgery, registering the end-expiratory skin surface to the real-time skin surface obtained by an optical imaging device to obtain the real-time displacement vector field of the skin surface; S4: using the full-process displacement vector fields of the skin and tumor surfaces and the real-time displacement vector field of the skin surface, calculating the real-time displacement vector field of the tumor surface using the inverse distance weighted interpolation method, and applying it to the end-expiratory tumor surface to obtain the real-time tumor location and morphology. This invention reduces the preoperative imaging radiation dose and the complexity of tumor tracking, contributing to precise lung radiotherapy.

Description

Technical Field

[0001] This invention belongs to the field of radiotherapy, and specifically relates to a method, system and storage medium for real-time estimation of the location and morphology of lung tumors. Background Technology

[0002] Radiation therapy (RT) is a treatment for malignant tumors that uses radiation (such as alpha, beta, and gamma rays produced by radioactive isotopes, as well as X-rays, electron beams, and proton beams produced by radiotherapy machines) to kill and damage cancer cells. RT works by destroying the cells in the irradiated area (target area), causing these cells to stop dividing and eventually die, thereby shrinking or eliminating the tumor.

[0003] Medical image 3D reconstruction is the process of converting medical image data into a 3D model, and it has wide applications in the field of medical imaging. Through medical image 3D reconstruction, doctors and researchers can more comprehensively understand and analyze the patient's anatomical structure, thereby providing more accurate information for diagnosis, surgical planning, and treatment.

[0004] 3D image registration is the process of aligning and matching 3D images from different datasets. It aligns images spatially to enable comparison, analysis, and integration of diverse image data. This involves steps such as feature extraction, feature matching, and transform model estimation. Registration yields a consistent image coordinate system, providing more accurate results.

[0005] A key technique in radiotherapy is maintaining precise tumor localization during treatment. However, respiration-induced movement and structural deformation can affect the accuracy of localization, leading to poor treatment outcomes. Increasing the volume of the target radiation area to compensate for tumor movement increases the risk of complications in normal tissues and limits the extent to which radiation doses can be increased to enhance tumor control. Therefore, during radiotherapy, it is usually necessary to estimate the location and morphology of the tumor to achieve precise localization.

[0006] Previously, preoperative 4D cone-beam CT and 4D MRI were typically used to guide intraoperative tumor localization, but differences remain compared to real-time intraoperative imaging. To reduce target volume while maintaining tumor control, recent research has developed various real-time navigation systems for radiotherapy.

[0007] The first approach utilizes respiratory gating to acquire external alternative signals driven from the patient's chest and abdominal surfaces, such as an active breathing coordinator, but this approach only considers the anterior-posterior direction, oversimplifying respiratory movements.

[0008] The second method employs a kV source-detector pair orthogonally placed to the LINAC gantry, enabling real-time X-ray fluoroscopy. This method results in additional imaging radiation dose and, due to the lack of sufficient information in the two-dimensional image, cannot effectively monitor target motion when the imaging beam is parallel to the motion.

[0009] The third method involves implanting markers in the target area for real-time tracking. This is an invasive procedure that requires additional imaging doses unless a dedicated detector is used.

[0010] The fourth approach involves building a model that maps skin surface movement to internal tumor movement. This method is non-invasive and requires no intraoperative radiographic imaging. It is divided into global models and patient-specific models. Global models utilize 4D CT scans of the general population to extract internal and external surrogate signals, but their accuracy is insufficient for individual patients. Patient-specific models use 4D CT data from individual patients to predict internal movement, at the cost of higher radiation exposure, as 4D CT scans typically generate 3-8 times more radiation than traditional 3D CT scans. Summary of the Invention

[0011] To address the issues of insufficient precision in radiotherapy for lung tumors during respiration and high radiation levels in preoperative imaging of existing patient-specific models, this invention provides a method, system, and storage medium for real-time estimation of the location and morphology of lung tumors. This method relies solely on 3D CT images from the end-expiratory and end-inspiratory phases to estimate the location and morphology of lung tumors during respiration.

[0012] To achieve the above objectives, the technical solution adopted by this invention is: a method for real-time estimation of the location and morphology of lung tumors, specifically including the following steps:

[0013] S1: Acquire lung CT images of the patient at the end of expiration and end of inspiration before surgery, and segment and reconstruct the skin and tumor surface respectively;

[0014] S2: The end-expiratory skin and tumor surface obtained in S1 are registered to the end-inspiratory skin and tumor surface using a non-rigid registration algorithm to obtain the full-process displacement vector field of the skin and tumor surface;

[0015] S3: During the procedure, a non-rigid registration algorithm is used to register the end-expiratory skin surface obtained in S1 to the real-time skin surface obtained by the optical imaging device, thereby obtaining the real-time displacement vector field of the skin surface.

[0016] S4: Based on the full-process displacement vector field of the skin and tumor surface obtained in S2 and the real-time displacement vector field of the skin surface obtained in S3, the real-time displacement vector field of the tumor surface is calculated using the inverse distance weighted interpolation method, and applied to the tumor surface at the end of expiration to obtain the real-time tumor position and morphology.

[0017] Further, in S1, the step of segmenting and reconstructing the skin and tumor surfaces separately includes the following: segmenting the binary masks of the human body and the tumor from the end-expiratory and end-inspiratory CT images respectively; using the traveling cubes method with an isosurface threshold of 0.5 to perform three-dimensional reconstruction of the binary masks to obtain a three-dimensional mesh model of the skin and tumor surfaces containing edges and vertices; and performing a halving operation on the three-dimensional mesh model of the skin to retain only the front half of the torso in the front-back direction.

[0018] Furthermore, in S1, the human body binarization mask is obtained by sequentially binarizing, opening, extracting the largest connected region, and filling holes in the end-expiratory and end-inspiratory CT images, while the tumor binarization mask is obtained by manual delineation or segmentation using an AI deep learning model.

[0019] Furthermore, in S2, S3, and S4, the non-rigid registration method is required to be applicable to the case where the registration source is a three-dimensional mesh, the registration target is a three-dimensional point cloud, and the registration source and the registration target partially overlap. The displacement vector field is a matrix composed of the displacement vectors of each vertex of the three-dimensional mesh.

[0020] Furthermore, in S3, the optical imaging device is at least able to acquire the depth value of a pixel, and the real-time skin surface is a three-dimensional point cloud of the skin surface of the front half of the human torso.

[0021] Furthermore, in S4, the step of calculating the real-time displacement vector field of the tumor surface using the inverse distance weighted interpolation method includes the following: removing some outliers in the real-time displacement vector field of the skin surface, calculating the proportion field of the real-time displacement of the skin surface, calculating the proportion field of the real-time displacement of the tumor surface according to the inverse distance weighted average method, and multiplying it element-wise with the whole process displacement vector field of the tumor surface to obtain the real-time displacement vector field of the tumor surface.

[0022] Furthermore, the step of removing some outliers in the real-time displacement vector field of the skin surface includes the following: removing displacement vectors in the real-time displacement vector field of the skin surface and the displacement vector field of the entire process of the skin surface with an angle exceeding 15°, and removing displacement vectors in the real-time displacement vector field of the skin surface with a displacement vector magnitude exceeding 0.05.

[0023] Furthermore, the step of calculating the real-time displacement ratio field of the skin surface includes the following: dividing the magnitude of each vector of the real-time skin surface displacement vector field by the magnitude of the corresponding vector of the whole-process displacement vector field of the skin surface, and combining them into a vector in the original order, wherein the vector is the real-time displacement ratio field of the skin surface.

[0024] Furthermore, the step of calculating the real-time displacement ratio field of the tumor surface according to the distance reciprocal weighted average method includes the following: for each vertex of the three-dimensional model of the tumor surface at the end of expiration, calculate the distance from it to the point corresponding to each vector in the real-time displacement vector field of the skin surface in the real-time three-dimensional point cloud of the skin surface, and use the reciprocal of the distance as the weight to perform a weighted average of the real-time displacement ratio field of the skin surface as the real-time displacement ratio of the vertex. The displacement ratios of all vertices constitute the real-time displacement ratio field of the tumor surface.

[0025] Furthermore, in S4, the step of applying the real-time displacement vector field of the tumor surface to the tumor surface at the end of expiration to obtain the real-time tumor position and morphology includes the following: adding the coordinate vectors of the three-dimensional mesh vertices of the tumor surface at the end of expiration to the corresponding vectors in the real-time displacement vector field of the tumor surface, while keeping the edge relationships between each vertex unchanged, to obtain the real-time tumor position and morphology.

[0026] This invention also proposes a real-time lung tumor location and morphology estimation system, comprising:

[0027] Preoperative CT image acquisition unit: used to acquire the end-expiratory and end-inspiratory CT image sequences;

[0028] Preoperative 3D reconstruction unit: used to segment and reconstruct a 3D mesh model of the skin and tumor surface at the end of expiration and inspiration from the CT images of the end-expiration and end-inspiration sequences;

[0029] Preoperative registration unit: used to non-rigidly register the three-dimensional mesh model of the skin and tumor surface at the end of expiration to the three-dimensional mesh model of the skin and tumor surface at the end of inspiration;

[0030] Intraoperative skin image acquisition unit: used to communicate with the optical imaging device and acquire real-time three-dimensional images of the skin surface;

[0031] Intraoperative registration unit: used to non-rigidly register the three-dimensional mesh model of the skin surface at the end of expiration to the real-time three-dimensional image of the skin surface;

[0032] Intraoperative tumor location and morphology estimation unit: used to calculate the real-time tumor location and morphology based on the lung tumor location and morphology real-time estimation method;

[0033] Visualization unit: used to display the end-expiratory, end-inspiratory, and real-time skin and tumor surface of the preoperative 3D reconstruction unit, the preoperative registration unit, the intraoperative registration unit, and the intraoperative tumor location and morphology estimation unit, as well as to display the real-time 3D image of the skin surface acquired by the intraoperative skin image acquisition unit.

[0034] The present invention also proposes a computer-readable storage medium, characterized in that the computer-readable storage medium stores instructions that, when executed on a processing component of a computer, cause the processing component to perform a tumor location and morphology estimation method.

[0035] The beneficial effects of this invention are as follows: Compared with existing technologies, this invention, as a non-invasive solution, can construct a lung tumor motion mapping model based on inverse distance weighted interpolation using only 3D CT scans taken at the end of expiration and inspiration before surgery. This enables 4D visualization of the location and morphology of lung tumors with higher efficiency and compatibility. Furthermore, it requires no additional radiation dose during treatment, reducing the radiation dose during pre-treatment 3D CT image acquisition and lowering hardware costs. This invention can be used for real-time tumor location and morphology estimation in precise radiotherapy for lung tumors, offering advantages such as non-invasive operation, reduced radiation dose, lower equipment costs, and improved estimation accuracy. Attached Figure Description

[0036] Figure 1 This is an overall flowchart of a method for real-time estimation of the location and morphology of lung tumors provided in an embodiment of the present invention.

[0037] Figure 2 This is a flowchart illustrating a real-time estimation method for the location and morphology of lung tumors provided in an embodiment of the present invention.

[0038] Figure 3 This is a framework diagram of a real-time lung tumor location and morphology estimation system provided in an embodiment of the present invention.

[0039] Figure 4 This is a flowchart of a method for reconstructing a three-dimensional image of the skin surface from a CT sequence, provided by an embodiment of the present invention.

[0040] Figure 5 This is a flowchart of a method for reconstructing a three-dimensional image of a tumor surface from a sequence CT scan, provided by an embodiment of the present invention.

[0041] Figure 6 This is an example of a three-dimensional point cloud of the skin surface acquired by the intraoperative skin image acquisition unit of a real-time estimation system for the location and morphology of a lung tumor, provided in an embodiment of the present invention.

[0042] Figure 7 This is a comparison image of the location and morphology of a lung tumor before and after the application of a real-time estimation method for the location and morphology of a lung tumor provided in an embodiment of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings and examples. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0044] like Figure 3 As shown, the real-time lung tumor location and morphology estimation system includes seven units: preoperative CT image acquisition unit, preoperative three-dimensional reconstruction unit, preoperative registration unit, intraoperative skin image acquisition unit, intraoperative registration unit, intraoperative tumor location and morphology estimation unit, and visualization unit.

[0045] Among them, the preoperative CT image acquisition unit is used to acquire CT images at the end of expiration and end of inspiration, and load the CT image files into the system.

[0046] The preoperative 3D reconstruction unit is used to segment and reconstruct a 3D mesh model of the skin and tumor surface at the end of expiration and end of inspiration from the loaded sequence of CT images using the traveling cubes method.

[0047] The preoperative registration unit is used to non-rigidly register the three-dimensional mesh model of the skin and tumor surface at the end of expiration to the three-dimensional mesh model of the skin and tumor surface at the end of inspiration using a non-rigid registration algorithm, and load the displacement vector field obtained from the registration into the system.

[0048] The intraoperative skin image acquisition unit is used to communicate with the optical imaging device and acquire real-time three-dimensional data of the skin surface, preprocess the three-dimensional data into a three-dimensional point cloud, and load it into the system.

[0049] The intraoperative registration unit is used to continuously use a non-rigid registration algorithm throughout the entire intraoperative process to non-rigidly register the three-dimensional mesh model of the skin and tumor surface at the end of expiration to the loaded real-time three-dimensional point cloud of the skin surface, and to load the displacement vector field obtained from the registration into the system.

[0050] The intraoperative tumor location and morphology estimation unit is used to calculate the real-time tumor location and morphology based on the displacement vector field and the real-time lung tumor location and morphology estimation method in the system.

[0051] Visualization Unit: Used to display all three-dimensional images of the preoperative 3D reconstruction unit, preoperative registration unit, intraoperative skin image acquisition unit, intraoperative registration unit, and intraoperative tumor location and morphology estimation unit on the system user interface.

[0052] The method flow of this invention embodiment is as follows: Figure 1 As shown, the process with image examples is as follows: Figure 2 As shown.

[0053] The following is through Figure 1 Steps S1, S2, S3, and S4 describe the specific steps of the embodiment:

[0054] Step S1: Acquire CT images at the end of expiration and end of inspiration before surgery, and segment and reconstruct the skin and tumor surface respectively.

[0055] This step is by Figure 3 The preoperative CT image acquisition unit, preoperative 3D reconstruction unit, and visualization unit shown are now complete.

[0056] Specifically, the system user interface allows users to select sequence CT image files in dcm, mha, mhd, or nrrd format, or select a folder containing dcm files of all sequence CT images. The system loads the files from storage into memory via ITK and reads the CT image metadata and images.

[0057] Specifically, the process of segmenting and reconstructing the skin from CT images is as follows: Figure 4 As shown, the process first involves binarizing all pixel values ​​of the image in memory, opening them, extracting the largest sub-connected region, and filling holes.

[0058] Furthermore, the processed image is used as input to the traveling cubes face drawing 3D reconstruction algorithm. The isosurface value of the traveling cubes algorithm is set to 0.5. The traveling cubes algorithm is then executed to obtain a 3D mesh of the skin surface.

[0059] It should be noted that a 3D mesh is different from a 3D point cloud. A 3D mesh is a surface 3D image that includes a set of vertices and a set of edges.

[0060] Furthermore, the latter half of the three-dimensional mesh on the skin surface is removed, preferably simplified to 10,000 faces, and then retained in memory.

[0061] Furthermore, the reconstructed 3D mesh of the skin surface is visualized using the VTK toolkit. Figure 2 and Figure 4 Images of them are displayed in all of them.

[0062] Specifically, the process of segmenting and reconstructing the tumor from CT images is as follows: Figure 5 As shown, the system first uses an AI deep learning model to read the CT image into memory and processes it to obtain a binary mask of the tumor. The mask image is then stored in memory, and the system displays the mask and the CT image in the system user interface.

[0063] Furthermore, the user determines whether the AI ​​deep learning model has correctly segmented the tumor mask. If it is incorrect, the user uses 3D Slicer software to read in the CT image and uses the delineation function to manually segment the tumor binarization mask and save the mask image file to storage. Then the system loads it from storage into memory.

[0064] Furthermore, the binarized mask image is used as input to the traveling cubes algorithm, with the isosurface value of the traveling cubes algorithm set to 0.5. The traveling cubes algorithm is then executed to obtain a three-dimensional mesh of the tumor surface, which is then stored in memory.

[0065] Furthermore, the reconstructed 3D mesh of the tumor surface was visualized using the VTK toolkit. Figure 2 and Figure 5 Images of them are displayed in all of them.

[0066] Step S2: Use a non-rigid registration algorithm to register the skin and tumor surface at the end of expiration to the skin and tumor surface at the end of inspiration, and obtain the displacement vector field of the skin and tumor surface throughout the process.

[0067] This step is by Figure 3 The preoperative registration and visualization units shown are now complete.

[0068] Specifically, after acquiring CT images of the end-expiratory and end-inspiratory phases in step S1 and reconstructing the three-dimensional mesh of the skin and tumor surface at the end-expiratory and end-inspiratory phases:

[0069] A non-rigid registration algorithm is used to register the three-dimensional image of the skin surface at the end of expiration to the three-dimensional image of the skin surface at the end of inhalation, thereby obtaining the displacement vector field of the skin throughout the process, which is then stored in memory.

[0070] A non-rigid registration algorithm was used to register the three-dimensional image of the tumor surface at the end of expiration to the three-dimensional image of the tumor surface at the end of inspiration, thereby obtaining the displacement vector field of the tumor throughout the entire process, which was then stored in memory.

[0071] Specifically, a non-rigid registration algorithm is used to iteratively minimize the energy function containing alignment, regularization, and rotation terms to find the optimal transition matrix, so that the vertex set of the three-dimensional mesh of the skin or tumor surface at the end of expiration corresponds optimally with the skin or tumor surface at the end of inhalation after the transformation of the transition matrix.

[0072] Furthermore, displacement vectors are extracted from this optimal transition matrix, and all displacement vectors of the vertex set of the skin or tumor surface at the end of exhalation constitute the full-process displacement vector field of the skin or tumor surface.

[0073] The displacement vector field of the skin or tumor surface throughout the entire process is represented in the visualization unit by drawing line segments of different gray levels between corresponding vertices of the source 3D mesh and the target 3D mesh. Figure 2 Images are displayed.

[0074] Step S3: The skin surface at the end of expiration is registered to the real-time skin surface obtained by the optical imaging device during the operation using a non-rigid registration algorithm, so as to obtain the real-time displacement vector field of the skin surface and store it in memory.

[0075] This step is by Figure 3 The intraoperative skin image acquisition unit, intraoperative registration unit, and visualization unit shown are now complete.

[0076] Specifically, during radiotherapy, optical imaging devices such as structured light, binocular cameras, or time-of-flight cameras can be selectively used to photograph the anterior half of the patient's torso to obtain a three-dimensional image of the skin surface of the anterior half of the torso, and the three-dimensional image file is saved to a storage medium shared by the imaging device and the system.

[0077] Furthermore, the system reads 3D image files from storage, processes them, converts them into 3D point cloud files in PLY format, and loads them into memory.

[0078] Preferably, the real-time three-dimensional point cloud of the skin surface is subjected to noise reduction processing.

[0079] Furthermore, the VTK toolkit is used to visualize the 3D point cloud of the skin surface obtained by the optical imaging device. Figure 6 This is an example.

[0080] Furthermore, a non-rigid registration algorithm is used to iteratively minimize the energy function containing alignment, regularization, and rotation terms to find the optimal transition matrix. This ensures that the vertex set of the three-dimensional mesh on the skin or tumor surface at the end of expiration corresponds optimally with the three-dimensional point cloud of the skin surface captured by the optical device after the transformation of the transition matrix.

[0081] Preferably, since the three-dimensional images captured by optical devices may have field-of-view defects and noise, the non-rigid registration algorithm used must have strong robustness to adapt to noise and partial overlap.

[0082] In addition, the non-rigid registration algorithm used must be able to register to point clouds rather than meshes.

[0083] In this embodiment, the Non-rigid ICP registration algorithm is used.

[0084] Furthermore, displacement vectors are extracted from this optimal transition matrix, and all displacement vectors of the skin surface vertex set constitute the real-time displacement vector field of the skin surface.

[0085] The real-time displacement vector field of the skin surface is represented in the visualization unit by drawing line segments of different gray levels between corresponding vertices of the source 3D mesh and the target 3D point cloud. Figure 2 Images are displayed.

[0086] Step S4: Based on the displacement vector field of the whole process and the real-time displacement vector field of the skin surface, the real-time displacement vector field of the tumor surface is calculated using the distance reciprocal weighted interpolation method, and applied to the tumor surface at the end of expiration to obtain the real-time tumor position and morphology.

[0087] This step is by Figure 3 The intraoperative tumor location and morphology estimation unit and visualization unit shown are completed.

[0088] Specifically, the skin surface displacement vector field obtained from step S2, which is stored in memory, is... ( (representing the set of real numbers), and the real-time displacement vector field of the skin surface obtained in step S3. (where n is the number of vertices in the 3D mesh on the skin surface at the end of expiration), the real-time displacement ratio field of the skin surface is calculated using the following formula. And keep it in memory:

[0089]

[0090] Furthermore, the influence of abnormal displacement vectors in the real-time displacement vector field on the skin surface is eliminated.

[0091] Specifically, the following elimination method is used:

[0092] Vertex coordinates are generally expressed in millimeters. If the magnitude of a vector in the real-time displacement vector field of the skin surface is less than 0.05, or if the angle between the vector and the corresponding displacement vector in the total displacement vector field of the skin surface exceeds 15°, the displacement ratio corresponding to that displacement vector is retrieved from the real-time displacement ratio field r of the skin surface in memory. skin Remove from memory and store the 3D mesh vertex set V of the end-expiratory skin surface in memory. skin The corresponding vertex in the data is removed.

[0093] Specifically, the index set I of the displacement vector field that should be retained rather than discarded can be represented as:

[0094]

[0095] Furthermore, the 3D mesh vertex set of the skin surface after expiration was removed. Real-time displacement ratio field of skin surface The following formula is used to obtain (where n′ is the number of points in the 3D mesh vertex set of the skin surface at the end of expiration after removal), and it is stored in memory:

[0096] V′ skin =[V skin [i], i∈I];

[0097] r′ skin =[r skin [i], i∈I];

[0098] Furthermore, based on the principle that the displacement of points on the skin surface affects the displacement of points on the tumor surface and that the distance between the two is inversely proportional, the real-time displacement vector field of the tumor surface is estimated using the inverse distance weighted interpolation method.

[0099] Specifically, based on the vertex set of the three-dimensional mesh of the tumor surface at the end of expiration obtained in step S1 in memory... The 3D mesh vertex set V′ of the skin surface at the end of expiration after removal skin (where m is the number of vertices in the 3D mesh on the tumor surface at the end of expiration), the real-time displacement scale field on the tumor surface is calculated using the following distance inverse weighted average formula.

[0100]

[0101] Furthermore, the real-time displacement vector field of the tumor surface should be a matrix obtained by element-wise multiplication of the real-time displacement proportional field of the tumor surface and the displacement vector field of the entire process of the tumor surface. Therefore, based on the displacement vector field of the entire process of the tumor surface obtained in step S2 in memory... The real-time displacement vector field on the tumor surface is calculated using the following formula.

[0102] TDVF t [i] = TDVF eoi [i]·r tumor [i];

[0103] Furthermore, by applying the real-time displacement vector field of the tumor surface to the vertex set of the three-dimensional mesh of the tumor surface at the end of expiration, the vertex set of the real-time three-dimensional mesh of the tumor surface can be obtained. Therefore, based on the vertex set V of the three-dimensional mesh of the tumor surface at the end of expiration in memory... tumor The vertex set of the real-time 3D mesh on the tumor surface is calculated using the following formula.

[0104]

[0105] Furthermore, the edge set of the end-tidal tumor surface 3D mesh is copied to the edge set of the real-time tumor surface 3D mesh to obtain the complete end-tidal tumor surface 3D mesh, which is then stored in memory.

[0106] Furthermore, by using the VTK toolkit to visualize the real-time 3D point cloud of the skin surface and the 3D mesh of the tumor surface at the end of expiration, users can observe the real-time location and morphology of the tumor.

[0107] It should be noted that this invention simulates the real-time tumor surface by calculating the real-time displacement vector field of the tumor surface and applying it to the tumor surface at the end of expiration, causing it to shift and deform, thereby achieving the effect of real-time estimation of tumor location and morphology.

[0108] Figure 7 The tumor location and morphology of this embodiment are shown. The left figure shows the end-tidal tumor and real-time tumor location and morphology before applying the method of the present invention, and the right figure shows the tumor after applying the method of the present invention. It can be seen that after the end-tidal tumor surface is displaced and deformed by the method of the present invention, the real-time tumor location and morphology can be basically restored.

[0109] In this embodiment, the Dice coefficient of the end-tidal tumor and the real-time tumor before applying the method of the present invention was 0.674, and after applying the present invention, it was 0.878. The larger the Dice coefficient, the more overlap there is. The calculation method of the Dice coefficient (DSC) of graph A and graph B is shown in the following formula:

[0110]

[0111] Therefore, by applying this invention, the accuracy of tumor location and morphology estimation in the embodiments is greatly improved.

[0112] It should be noted that, in the examples of this invention, Figure 7 The lung surface shown is obtained through a three-dimensional reconstruction method of the tumor surface, similar to step S1. Its purpose is to facilitate user observation. Compared with the skin and tumor surface, the lung surface is not necessary for the real-time location and morphology estimation of the tumor. Therefore, the three-dimensional reconstruction and visualization of the lung surface is an optional step.

[0113] In summary, the real-time tumor location and morphology estimation method and system proposed in this invention can achieve relatively accurate tumor location and morphology estimation based solely on preoperative end-expiratory and end-inspiratory CT images and three-dimensional skin surface images captured by intraoperative optical equipment. Therefore, compared with previous methods, this invention achieves comparable accuracy while solving the problems of high preoperative imaging radiation dose, the need for intraoperative radiographic imaging and invasive operations, and high hardware costs. When applied to lung tumor tracking in image-guided radiotherapy, it can improve the accuracy of the location and size of the radiotherapy target area, thereby improving patient prognosis.

[0114] The real-time tumor location and morphology estimation system provided in this invention uses Qt software running on a Windows system for its user interface. Therefore, in addition to the computer, the only external device is an optical imaging device, and the two share the storage medium via a USB data cable. As a result, the system is easy to operate and has low cost.

[0115] This invention also provides a computer-readable storage medium storing instructions that, when executed on a computer's processing component, cause the processing component to perform... Figures 1 to 5 Any of the methods described.

[0116] This invention also provides a computer program product containing instructions, which, when run on a computer, causes the computer to perform... Figures 1 to 5 Any of the methods described.

[0117] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0118] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for real-time estimation of the location and morphology of lung tumors, characterized in that, Specifically, the following steps are included: S1: Acquire lung CT images of the patient at the end of expiration and end of inspiration before surgery, and segment and reconstruct the skin and tumor surface respectively; S2: The end-expiratory skin and tumor surface obtained in S1 are registered to the end-inspiratory skin and tumor surface using a non-rigid registration algorithm to obtain the full-process displacement vector field of the skin and tumor surface; S3: During the procedure, a non-rigid registration algorithm is used to register the end-expiratory skin surface obtained in S1 to the real-time skin surface obtained by the optical imaging device, thereby obtaining the real-time displacement vector field of the skin surface. S4: Based on the full-process displacement vector field of the skin surface obtained in S2 and the real-time displacement vector field of the skin surface obtained in S3, calculate the real-time displacement ratio field of the skin surface. Based on the vertex set of the three-dimensional mesh of the tumor surface at the end of expiration, the vertex set of the three-dimensional mesh of the skin surface at the end of expiration, and the real-time displacement ratio field of the skin surface, calculate the real-time displacement ratio field of the tumor surface using the distance reciprocal weighted average formula. Multiply the real-time displacement ratio field of the tumor surface by the full-process displacement vector field of the tumor surface element by element to obtain the real-time displacement vector field of the tumor surface; and apply it to the tumor surface at the end of expiration to obtain the real-time tumor position and morphology.

2. The method for real-time estimation of lung tumor location and morphology according to claim 1, characterized in that, In S1, the steps of segmenting and reconstructing the skin and tumor surfaces respectively include the following: segmenting the binary masks of the human body and the tumor from the end-expiratory and end-inspiratory CT images respectively; using the traveling cubes method with an isosurface threshold of 0.5 to perform three-dimensional reconstruction of the binary masks to obtain a three-dimensional mesh model of the skin and tumor surfaces containing edges and vertices; and performing a halving operation on the three-dimensional mesh model of the skin to retain only the front half of the torso in the front-back direction.

3. The method for real-time estimation of lung tumor location and morphology according to claim 2, characterized in that, In S1, the human body binarization mask is obtained by sequentially binarizing, opening, extracting the largest connected region, and filling holes in the end-expiratory and end-inspiratory CT images. The tumor binarization mask is obtained by manual drawing or segmentation by an AI deep learning model.

4. The method for real-time estimation of lung tumor location and morphology according to claim 1, characterized in that, In S2 and S3, the non-rigid registration algorithm is required to be applicable to the case where the registration source is a three-dimensional mesh, the registration target is a three-dimensional point cloud, and the registration source and the registration target partially overlap. The displacement vector field is a matrix composed of the displacement vectors of each vertex of the three-dimensional mesh.

5. The method for real-time estimation of lung tumor location and morphology according to claim 1, characterized in that, In S3, the optical imaging device is able to acquire at least the depth value of a pixel, and the real-time skin surface is a three-dimensional point cloud of the skin surface of the front half of the human torso.

6. The method for real-time estimation of lung tumor location and morphology according to claim 1, characterized in that, In S4, the steps for calculating the real-time displacement vector field of the tumor surface using the inverse distance weighted interpolation method include the following: Some outliers in the real-time displacement vector field of the skin surface are removed, and the real-time displacement ratio field of the skin surface is calculated. The real-time displacement ratio field of the tumor surface is calculated according to the distance reciprocal weighted average method and multiplied element-wise with the whole-process displacement vector field of the tumor surface to obtain the real-time displacement vector field of the tumor surface.

7. The method for real-time estimation of lung tumor location and morphology according to claim 6, characterized in that, The step of removing outliers from the real-time displacement vector field of the skin surface includes the following: Displacement vectors whose angle between the corresponding displacement vectors in the real-time displacement vector field and the whole-process displacement vector field of the skin surface exceeds 15° are removed. Displacement vectors whose displacement vector magnitude exceeds 0.05 in the real-time displacement vector field of the skin surface are also removed.

8. The method for real-time estimation of lung tumor location and morphology according to claim 7, characterized in that, The steps for calculating the real-time displacement proportional field of the skin surface include the following: Divide the magnitude of each vector in the real-time skin surface displacement vector field by the magnitude of the corresponding vector in the full-process skin surface displacement vector field, and combine them in the original order to form a vector, which is the real-time displacement ratio field of the skin surface.

9. The method for real-time estimation of lung tumor location and morphology according to claim 8, characterized in that, The step of calculating the real-time displacement proportional field of the tumor surface using the inverse distance weighted average method includes the following: For each vertex of the three-dimensional model of the tumor surface at the end of expiration, the distance from it to the point in the three-dimensional point cloud of the real-time skin surface corresponding to each vector in the real-time displacement vector field of the skin surface is calculated, and the real-time displacement ratio of the real-time displacement ratio of the skin surface is weighted according to the reciprocal of the distance as the weight, and the displacement ratios of all vertices constitute the real-time displacement ratio field of the tumor surface.

10. The method for real-time estimation of lung tumor location and morphology according to claim 1, characterized in that, In S4, the step of applying a real-time displacement vector field of the tumor surface to the tumor surface at the end of expiration to obtain the real-time tumor location and morphology includes the following: The real-time tumor position and shape are obtained by adding the coordinate vectors of the three-dimensional mesh vertices on the tumor surface at the end of expiration to the corresponding vectors in the real-time displacement vector field of the tumor surface, while keeping the edge relationships between the vertices unchanged.

11. A real-time lung tumor location and morphology estimation system, characterized in that, include: Preoperative CT image acquisition unit: used to acquire the end-expiratory and end-inspiratory CT image sequences; Preoperative 3D reconstruction unit: used to segment and reconstruct a 3D mesh model of the skin and tumor surface at the end of expiration and inspiration from the CT images of the end-expiration and end-inspiration sequences; Preoperative registration unit: used to non-rigidly register the three-dimensional mesh model of the skin and tumor surface at the end of expiration to the three-dimensional mesh model of the skin and tumor surface at the end of inspiration; Intraoperative skin image acquisition unit: used to communicate with the optical imaging device and acquire real-time three-dimensional images of the skin surface; Intraoperative registration unit: used to non-rigidly register the three-dimensional mesh model of the skin surface at the end of expiration to the real-time three-dimensional image of the skin surface; Intraoperative tumor location and morphology estimation unit: used to calculate the real-time tumor location and morphology according to the lung tumor location and morphology real-time estimation method according to any one of claims 1-10; Visualization unit: used to display the end-expiratory, end-inspiratory, and real-time skin and tumor surface of the preoperative 3D reconstruction unit, the preoperative registration unit, the intraoperative registration unit, and the intraoperative tumor location and morphology estimation unit, as well as to display the real-time 3D image of the skin surface acquired by the intraoperative skin image acquisition unit.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a computer's processing component, cause the processing component to perform the real-time lung tumor location and morphology estimation method according to any one of claims 1 to 10.

Images