Real-time three-dimensional tumor tracking method and apparatus, and computer device and storage medium

Abstract

The present application relates to a real-time three-dimensional (3D) tumor tracking method and apparatus, and a computer device and a storage medium. The method comprises: acquiring a moving image and a reference image of a tumor site; performing hybrid data enhancement on the moving image and the reference image, so as to obtain a 3D CT image that represents each respiratory phase, and a segmented moving image; projecting the 3D CT image by using a ray casting algorithm, so as to generate a 2D DRR at a corresponding moment; and inputting the 2D DRR into a trained tumor tracking network, outputting a 3D deformation field of the 2D DRR by means of the tumor tracking network, and making the 3D deformation field act on the reference image, so as to generate a predicted segmented tumor image of the 2D DRR at the corresponding moment. In the embodiments of the present application, tumor tracking is performed on the basis of a single-angle X-ray image, such that the radiation dose is greatly reduced, high-precision tumor tracking performed at a low dose is realized, and the tumor tracking speed is significantly increased.

Description

Three-dimensional tumor real-time tracking method, device, computer equipment and storage medium Technical Field

[0001] The present application belongs to the field of medical image processing technology, and in particular relates to a three-dimensional tumor real-time tracking method, apparatus, computer equipment, and storage medium. Background Art

[0002] The goal of radiotherapy is to maximize the elimination of tumor cells while protecting surrounding normal tissues and organs. However, during radiotherapy, the shape and position of organs and tumors may change due to respiratory and cardiac motion, thus affecting the accuracy of radiotherapy. To ensure that the irradiation field closely follows the target while avoiding damage to surrounding normal tissues and organs, it is necessary to rapidly capture images during radiotherapy and register them with the images used to develop the radiotherapy plan. This allows for the real-time location of the tumor during radiotherapy and real-time tracking of tumor cells.

[0003] Tumor tracking methods in the prior art generally include the following two methods:

[0004] 1. Tracking using markers; this method includes both internal and external markers. Internal markers are implanted directly into the tumor or surrounding tissue, allowing for real-time tumor location information to be acquired through continuous imaging, resulting in high accuracy. However, this method is invasive and can easily lead to problems such as pneumothorax and immune rejection. External markers use an external surrogate as a replacement for the internal marker. This method requires establishing a relationship model between the external surrogate and the tumor. However, as the respiratory search and radiotherapy process progress, this relationship model may change, leading to inaccurate positioning and severely impacting the accuracy of radiotherapy.

[0005] 2. Tumor localization using image registration: This method automatically propagates tumor contours from pre-radiotherapy images to in-radiotherapy images by acquiring a mapping deformation field between 2D projection images and 3D images. This method can be categorized as model-based registration or data-driven registration. Model-based registration attempts to utilize dimensionality reduction models based on B-Spline or PCA to extract key coefficients or components, establish a patient space based on specific priors, and then complete the lung search model through iterative optimization. However, this method typically requires iterative optimization of each image to be registered, resulting in slow registration speed and failing to meet the real-time requirements of tumor localization. Data-driven registration methods, provided that the model has been trained with a large amount of data, can be directly applied during the testing phase without retraining, thus significantly reducing registration time. However, the 2D images used in this method typically require multiple X-ray or CBCT projections captured from multiple angles, resulting in high radiation doses, potential harm to the patient, and prolonged registration time.

[0006] Summary of the Invention

[0007] The present application provides a three-dimensional tumor real-time tracking method, apparatus, computer equipment and storage medium, which aims to solve at least one of the above-mentioned technical problems in the prior art to a certain extent.

[0008] In order to solve the above problems, this application provides the following technical solutions:

[0009] A three-dimensional real-time tumor tracking method, comprising:

[0010] Acquire floating images and reference images of the tumor site;

[0011] performing hybrid data enhancement on the floating image and the reference image to obtain a 3D CT image and a floating segmentation image representing each respiratory phase;

[0012] Projecting the 3D CT image using a light projection algorithm to generate a 2D DRR image at a corresponding moment;

[0013] The 2D DRR image is input into a trained tumor tracking network, and the 3D deformation field of the 2D DRR image is output through the tumor tracking network. The 3D deformation field is applied to the reference image to generate a tumor prediction segmentation image at the corresponding moment of the 2D DRR image.

[0014] The technical solution adopted in the embodiment of the present application also includes: the floating image and the reference image of the tumor site are obtained by:

[0015] The end of exhalation is used as a floating image M CT , and use the remaining respiratory phases as reference images M seg .

[0016] The technical solution adopted in the embodiment of the present application further includes: performing hybrid data enhancement on the floating image and the reference image to obtain a 3D CT image and a floating segmented image representing each respiratory phase is specifically as follows:

[0017] The floating image M is registered using the Demons image registration algorithm. CT and the reference image M seg Perform registration in sequence to obtain the deformation φ between each respiratory phase 1,...i,j ;

[0018] From the deformation φ 1,...i,j Randomly select two of them and perform random weight superposition to obtain the phase deformation from the end of exhalation to the remaining respiratory phases;

[0019] The thin plate spline interpolation algorithm is used to simulate the setting of random control points and random movement distances to obtain the intra-phase deformation of a certain respiratory phase;

[0020] According to the inter-phase deformation and intra-phase deformation, a mixed deformation field φ is obtained. ybrid , and the mixed deformation field φ ybrid Acting on the end of exhalation, a 3D CT image S representing each exhalation phase is obtained. CT .

[0021] The technical solution adopted in the embodiment of the present application further includes: after projecting the 3D CT image using the light projection algorithm to generate a 2D DRR image at the corresponding moment, it also includes:

[0022] The 2D DRR image is corrected using a generative adversarial network to obtain a simulated X-ray image C with the style of a real X-ray image but the image content remains unchanged. proj .

[0023] The technical solution adopted in the embodiment of the present application also includes: the image correction of the 2D DRR image using the generative adversarial network is specifically as follows:

[0024] The generative adversarial network is a CUT network, which adopts contrastive learning of multi-level image blocks, takes 2D DRR images as the source domain and real X-ray images as the target domain, and uses a generator and a discriminator to convert the 2D DRR images into simulated X-ray images with the style of real X-ray images but the image content remains unchanged.

[0025] The technical solution adopted in the embodiment of the present application further includes: before inputting the 2D DRR image into the trained tumor tracking network, it also includes:

[0026] The simulated X-ray image C proj Input the tumor tracking network, and extract the simulated X-ray image C using the residual block through the tumor tracking network proj The features of the channel layer are used as the third dimension features to output a three-dimensional feature map;

[0027] The three-dimensional feature map is input into the 3D feature extraction network, which is a registration network based on the combination of Swintransformer and CNN. The 3D feature extraction network outputs a 3D deformation field φ under the joint action of the encoding and decoding mechanism and the self-attention mechanism. p ;

[0028] The 3D deformation field φ is transformed into p Acting on the reference image M seg , obtain the simulated X-ray image C proj The corresponding tumor prediction segmentation image P seg ;

[0029] Calculate the tumor prediction segmentation image P seg and floating segmented image S seg The maximum similarity between them is calculated, and the tumor tracking network is iteratively optimized according to the maximum similarity to obtain a trained tumor tracking network.

[0030] The technical solution adopted in the embodiment of the present application also includes: the tumor tracking network uses the tumor prediction segmentation image P seg and floating segmented image S seg The Dice similarity index and the MSE between the predicted deformation field and the true deformation field, as well as the smoothness constraint of the deformation field are used as the loss function: L=λ1L Dice (S seg P seg )+λ2L MSE (φ hybrid ,φ p )+λ3L Reg (φ p )

[0031] In the above formula, n represents the number of categories in the image, k represents the kth category of the image, represents all elements in the entire deformation field, and λ1, λ2, and λ3 represent L Dice , L MSE , L Reg The weight of .

[0032] Another technical solution adopted in the embodiment of the present application is: a three-dimensional tumor real-time tracking method and device, comprising:

[0033] Image acquisition module: used to obtain floating images and reference images of the tumor site;

[0034] A data enhancement module is used to perform hybrid data enhancement on the floating image and the reference image to obtain a 3D CT image and a floating segmentation image representing each respiratory phase;

[0035] Image projection module: used to project the 3D CT image using a light projection algorithm to generate a 2D DRR image at the corresponding moment;

[0036] Tumor prediction module: used to input the 2D DRR image into the trained tumor tracking network, output the 3D deformation field of the 2D DRR image through the tumor tracking network, apply the 3D deformation field to the reference image, and generate a tumor prediction segmentation image at the corresponding moment of the 2D DRR image.

[0037] Another technical solution adopted by the embodiment of the present application is: a computer device, the computer device includes a processor and a memory coupled to the processor, wherein:

[0038] The memory stores program instructions for implementing the three-dimensional tumor real-time tracking method;

[0039] The processor is configured to execute the program instructions stored in the memory to control a three-dimensional tumor real-time tracking method.

[0040] Another technical solution adopted by the embodiment of the present application is: a storage medium storing program instructions executable by a processor, wherein the program instructions are used to execute the three-dimensional real-time tumor tracking method.

[0041] Compared with the prior art, the beneficial effects produced by the embodiments of the present application are as follows: the three-dimensional tumor real-time tracking method, device, computer equipment and storage medium of the embodiments of the present application obtain 3D CT images and floating segmented images representing various respiratory phases of the lungs by performing mixed data enhancement on floating images and reference images, project the 3D CT images using a ray projection algorithm, generate 2DDRR images at the corresponding moments, input the 2DDRR images into the trained tumor tracking network, and the tumor tracking network uses a network framework that combines Swin Transformer and CNN to achieve tumor tracking based on a single X-ray image. The embodiments of the present application track tumors based on single-angle X-ray images, which greatly reduces the radiation dose and can achieve low-dose, high-precision 3D tumor tracking in a shorter time, which is beneficial to image-guided surgery and radiotherapy, while simplifying the hardware of the tomographic imaging system. At the same time, the embodiments of the present application use a combination of Swin Transformer and CNN for tumor tracking, which significantly improves the speed of tumor tracking while retaining the self-attention mechanism to obtain higher accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] FIG1 is a flow chart of a three-dimensional real-time tumor tracking method according to an embodiment of the present application;

[0043] FIG. 2 is a diagram of a floating image M in an embodiment of the present application. CT and the reference image S seg Flowchart for hybrid data augmentation;

[0044] FIG3 is a schematic diagram of an image correction framework according to an embodiment of the present application;

[0045] FIG4 is a schematic diagram of a tumor tracking network structure according to an embodiment of the present application;

[0046] FIG5 is a schematic diagram of a tumor registration experiment for data of a patient with a tumor from the end of exhalation to the end of inspiration;

[0047] FIG6 is a schematic diagram of experimental results for three types of tumors: large, medium, and small;

[0048] FIG7 is a schematic diagram showing the results of tumor localization by bringing real CBCT projections into the tumor tracking network;

[0049] FIG8 is a schematic diagram of the structure of a device for a three-dimensional real-time tumor tracking method according to an embodiment of the present application;

[0050] FIG9 is a schematic diagram of the structure of a computer device according to an embodiment of the present application;

[0051] FIG10 is a schematic diagram of the structure of a storage medium according to an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0053] The terms "first," "second," and "third" in this application are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of such features. In the description of this application, "multiple" means at least two, for example, two, three, etc., unless otherwise specifically defined. All directional indications in the embodiments of this application (such as up, down, left, right, front, back...) are only used to explain the relative positional relationship, movement, etc. between the components under a specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indications also change accordingly. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or computer device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products, or computer devices.

[0054] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0055] Please refer to FIG1 , which is a flow chart of a three-dimensional tumor real-time tracking method according to an embodiment of the present application. The three-dimensional tumor real-time tracking method according to an embodiment of the present application comprises the following steps:

[0056] S100: Acquire a floating image and a reference image of the tumor site;

[0057] In this step, medical image registration is to perform a certain geometric transformation on one image and map it to another image so that the relevant points in the two images are spatially consistent. The image to which the geometric transformation is applied is defined as the moving image, and the other fixed image is defined as the reference image. In this embodiment, a 4D CT image of the lungs is selected as the experimental object, and the end of expiration is used as the floating image M. CT , and use the remaining respiratory phases as reference images M seg , specifically, it may be a CT image of the heart or other parts of the body.

[0058] S110: Floating image M CT and the reference image M seg Perform hybrid data enhancement to obtain a 3DCT image S representing each respiratory phase CT and floating segmented image S seg ;

[0059] In this step, since real 2D / 3D medical images of the same moment are very scarce, the embodiment of the present application uses data enhancement to obtain a large number of 3D CT images representing various respiratory phases of the lungs. Specifically, as shown in FIG2 , the embodiment of the present application obtains a large number of 3D CT images representing various respiratory phases of the lungs. CT and the reference image M seg The flowchart for hybrid data enhancement includes the following steps:

[0060] S111: Using Demons image registration algorithm to register floating image M CT and the reference image M seg Perform registration in sequence to obtain the deformation φ between each respiratory phase 1,...i,j ;

[0061] S112: From deformation φ 1,...i,j Randomly select two of them and perform random weight superposition to obtain a large number of phase deformations from the end of exhalation to the remaining respiratory phases;

[0062] S113: using a thin plate spline interpolation algorithm to simulate setting random control points and random movement distances, thereby obtaining a large number of intra-phase deformations simulating a certain respiratory phase;

[0063] S114: A large number of mixed deformation fields φ are obtained based on inter-phase deformation and intra-phase deformationybrid , and the mixed deformation field φ ybrid Acting on the end of exhalation, a 3D CT image S representing each exhalation phase of the lungs is obtained. CT .

[0064] S120: Using the light projection algorithm to 3D CT image S CT Projection is performed to generate a 2DDRR (Digitally Reconstructed Radiograph) image at the corresponding moment;

[0065] In this step, the 3D CT image S is projected using a light projection algorithm. CT Projection is performed to generate a 2D digitally reconstructed image at the corresponding moment to simulate a real X-ray image.

[0066] S130: Use the generative adversarial network to perform image correction on the 2D DRR image to obtain a simulated X-ray image C with the style of a real X-ray image but the image content remains unchanged proj ;

[0067] In this step, in order to achieve 2D / 3D image alignment, it is necessary to project the data-enhanced 3D CT image SCT onto a 2D image by light projection to simulate a real X-ray image. However, there are still certain differences in noise and grayscale between 2DDRR and real X-ray images. In order to solve this problem, the embodiment of the present application uses GAN (Generative Adversarial Networks) to perform image correction on 2DDRR, so that the corrected 2DDRR has the style of a real X-ray image while maintaining the original image content unchanged. Specifically, the embodiment of the present application selects the CUT network (Contrastive Learning for Unpaired Image-to-Image Translation, an unpaired generative adversarial network based on contrastive learning), adopts contrastive learning of multi-level image blocks, and realizes unidirectional image correction through a generator and a discriminator. Specifically, as shown in Figure 3, it is a schematic diagram of the image correction framework of an embodiment of the present application, wherein DRR is a simulated two-dimensional image, X-ray is a real X-ray image, DRR is used as the source domain, and X-ray is used as the target domain. While saving resources, a generator and a discriminator are used to convert the DRR image into a simulated X-ray image with the style of a real X-ray image.

[0068] S140: Simulate X-ray image C proj Input the tumor tracking network and output the 3D deformation field φ through the tumor tracking network p , apply the 3D deformation field φp to the reference image Mseg , generate simulated X-ray image C proj The corresponding tumor prediction segmentation image P seg , and use the tumor prediction segmentation image P seg and floating segmented image S seg The tumor tracking network is iteratively optimized based on the maximum similarity between them to obtain a trained tumor tracking network;

[0069] In this step, the tumor tracking network uses the combined effect of the Swin Transformer network and the CNN network to achieve tumor tracking based on a single X-ray image. Specifically, as shown in Figure 4, it is a schematic diagram of the tumor tracking network structure of the embodiment of the present application. The tumor tracking network uses the dimensionality increase method to take the channel layer as the third dimension feature, transforming the 2D / 3D registration problem into a 3D / 3D registration problem. First, the residual block is used to extract the simulated X-ray image C proj The features of the 3D feature map are taken as the third dimension features, and the 3D feature map is output. The 3D feature map is then input into the 3D feature extraction network. The 3D feature extraction network is a registration network based on the combination of Swintransformer and CNN. Under the joint action of the encoding and decoding mechanism and the self-attention mechanism, the 3D deformation field φ is output. p ; Then, the 3D deformation field φ is transformed through the spatial transformation layer p Acting on the reference image M seg , to obtain the simulated X-ray image C proj The corresponding tumor prediction segmentation image P seg , and calculate the tumor prediction segmentation image P seg and floating segmented image S seg The maximum similarity between the two images is obtained, and the tumor tracking network is continuously iteratively optimized based on the maximum similarity to complete the training of the tumor tracking network. The embodiment of the present application uses the Swin Transformer and CNN to perform tumor tracking, which can achieve higher tumor tracking accuracy while retaining the self-attention mechanism and speeding up tumor tracking.

[0070] Furthermore, in the model training process, the embodiment of the present application uses the Dice similarity index between the predicted tumor segmentation image and the true floating segmentation image, the MSE (Mean Square Error) of the predicted deformation field and the true deformation field, and the smoothness constraint of the deformation field as the loss function: L=λ1L Dice (S seg P seg )+λ2L MSE (φ hybrid ,φ p )+λ3LReg (φ p ) (3)

[0071] In the above formula, n represents the number of categories in the image, k represents the kth category of the image, and λ1, λ2, and λ3 represent all elements in the entire deformation field. Dice , L MSE , L Reg The weights of , in the embodiment of the present application, are 1.0, 0.1, and 0.01 respectively.

[0072] S150: Input a single 2D X-ray image or 2D DRR image into a trained tumor tracking network, output a predicted 3D deformation field through the tumor tracking network, and apply the 3D deformation field to the reference image to obtain a 3D tumor image at the corresponding moment of the single 2D X-ray image or 2D DRR image.

[0073] In order to verify the feasibility and effectiveness of the embodiments of the present application, the following embodiments conducted a tumor registration experiment from the end of exhalation to the end of inspiration on the patient data with tumors. The experimental results are shown in Figure 5, where the odd columns are unregistered images and the even rows are registered images. Specifically, the first and second rows show the shapes of the lungs before and after registration (segmented images), and the third and fourth rows show the images before and after tumor registration. Experiments were also conducted from the end of exhalation to the end of inspiration on a dynamic lung phantom with three different tumor sizes. The specific experimental results are shown in Figure 6, which shows the registration results of large, medium and small tumors from left to right, where the odd columns are unregistered images and the even rows are registered images. It can be seen that both real patient and phantom experiments achieved good tumor tracking results.

[0074] In addition, we evaluated the phantom using real X-ray images. Specifically, Figure 7 shows the results of bringing real CBCT projections into the tumor tracking network for tumor localization. The left side (a) shows the result with CUT correction, and the right side (b) shows the result without CUT correction. From top to bottom are the target area delineation curves for large tumors, medium tumors, and small tumors, where the black outline is the gold standard, the white outline is the predicted value, and the crosses within the outline are the reference tumor centroid and the predicted tumor centroid, respectively. Ablation experiments with and without image correction showed that the difference between the tumor image prediction results obtained using the tumor tracking network proposed in this embodiment of the application and the tumor in the real CBCT projection drawn by the doctor is very small (the root mean square error of the tumor centroid is less than 1.5 mm), achieving a high tracking accuracy.

[0075] Based on the above, the three-dimensional tumor real-time tracking method of the embodiment of the present application obtains a 3D CT image and a floating segmentation image representing each respiratory phase of the lung by performing mixed data enhancement on the floating image and the reference image, and projects the 3D CT image using a ray projection algorithm to generate a 2DDRR image at the corresponding moment. The reconstructed image is corrected using the CUT network to obtain a simulated X-ray image with a real X-ray image style but the image content remains unchanged. The simulated image is then input into the tumor tracking network, and the tumor tracking network uses a network framework that combines Swin Transformer and CNN to achieve tumor tracking based on a single X-ray image. The embodiment of the present application performs tumor tracking based on single-angle X-ray images, which greatly reduces the irradiation dose and can achieve low-dose, high-precision 3D tumor tracking in a shorter time, which is beneficial to image-guided surgery and radiotherapy, while simplifying the hardware of the tomographic imaging system. At the same time, the embodiment of the present application uses a combination of Swin Transformer and CNN for tumor tracking, which significantly improves the speed of tumor tracking while retaining the self-attention mechanism to obtain higher accuracy.

[0076] Please refer to FIG8 , which is a schematic diagram of the structure of a three-dimensional tumor real-time tracking method and apparatus according to an embodiment of the present application. The three-dimensional tumor real-time tracking method and apparatus 40 according to the embodiment of the present application comprises:

[0077] Image acquisition module 41: used to acquire a floating image and a reference image of the tumor site;

[0078] Data enhancement module 42: used for performing hybrid data enhancement on the floating image and the reference image to obtain a 3D CT image and a floating segmentation image representing each respiratory phase;

[0079] Image projection module 43: used to project the 3D CT image using a light projection algorithm to generate a 2D DRR image at a corresponding moment;

[0080] Tumor prediction module 44: used to input the 2D DRR image into a trained tumor tracking network, output the 3D deformation field of the 2D DRR image through the tumor tracking network, apply the 3D deformation field to the reference image, and generate a tumor prediction segmentation image at the corresponding moment of the 2D DRR image.

[0081] Please refer to FIG9 , which is a schematic diagram of the computer device structure of an embodiment of the present application. The computer device 50 includes:

[0082] A memory 51 storing executable program instructions;

[0083] a processor 52 connected to the memory 51;

[0084] The processor 52 is used to call the executable program instructions stored in the memory 51 and execute the following steps: obtaining a floating image and a reference image of the tumor site; performing mixed data enhancement on the floating image and the reference image to obtain a 3D CT image and a floating segmentation image representing each respiratory phase; projecting the 3D CT image using a light projection algorithm to generate a 2D DRR image at the corresponding moment; inputting the 2D DRR image into a trained tumor tracking network, outputting a 3D deformation field of the 2D DRR image through the tumor tracking network, applying the 3D deformation field to the reference image, and generating a predicted tumor segmentation image at the corresponding moment of the 2D DRR image.

[0085] The processor 52 may also be referred to as a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip having signal processing capabilities. The processor 52 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor or any conventional processor.

[0086] Please refer to Figure 10, which is a schematic diagram of the structure of a storage medium in an embodiment of the present application. The storage medium in an embodiment of the present application stores program instructions 61 capable of implementing the following steps: acquiring a floating image and a reference image of the tumor site; performing hybrid data enhancement on the floating image and the reference image to obtain a 3D CT image and a floating segmentation image representing each respiratory phase; projecting the 3D CT image using a ray projection algorithm to generate a 2D DRR image at the corresponding moment; inputting the 2D DRR image into a trained tumor tracking network, outputting a 3D deformation field of the 2D DRR image through the tumor tracking network, applying the 3D deformation field to the reference image, and generating a predicted tumor segmentation image at the corresponding moment of the 2D DRR image. The program instructions 61 can be stored in the above-mentioned storage medium in the form of a software product, including several instructions for causing a computer device (which can be a personal computer, server, or network computer device, etc.) or a processor to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage media include: various media that can store program instructions, such as USB flash drives, mobile hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks or optical disks, or terminal computer devices such as computers, servers, mobile phones, and tablets. The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0087] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0088] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. The above is only an implementation method of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the content of the description and drawings of this application, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A three-dimensional real-time tumor tracking method, It is characterized in that include: Acquire floating images and reference images of the tumor site; Performing hybrid data enhancement on the floating image and the reference image to obtain a 3D CT image and a floating segmentation image representing each respiratory phase; Projecting the 3D CT image using a light projection algorithm to generate a 2D DRR image at a corresponding moment; The 2D DRR image is input into a trained tumor tracking network, and the 3D deformation field of the 2D DRR image is output through the tumor tracking network. The 3D deformation field is applied to the reference image to generate a tumor prediction segmentation image at the corresponding moment of the 2D DRR image.

2. The three-dimensional tumor real-time tracking method according to claim 1, It is characterized in that The method of obtaining the floating image and the reference image of the tumor site is specifically as follows: Set the end of exhalation as a floating image M CT , and the remaining respiratory phases are used as reference images M seg .

3. The three-dimensional tumor real-time tracking method according to claim 2, It is characterized in that The hybrid data enhancement of the floating image and the reference image to obtain the 3D CT image and the floating segmentation image representing each respiratory phase is specifically: The floating image M is registered using the Demons image registration algorithm. CT and the reference image M seg Perform registration in sequence to obtain the deformation φ between each respiratory phase 1,...i,j ; From the deformation φ 1,...i,j Randomly select two of them and perform random weight superposition to obtain the phase deformation from the end of exhalation to the remaining breathing phases; The thin plate spline interpolation algorithm is used to simulate the setting of random control points and random moving distances to obtain the intra-phase deformation of a certain respiratory phase; According to the inter-phase deformation and intra-phase deformation, a mixed deformation field φ is obtained. ybrid , and the mixed form Variation field ybrid Acting on the end of exhalation, a 3D CT image S representing each exhalation phase is obtained CT .

4. The three-dimensional tumor real-time tracking method according to any one of claims 1 to 3, It is characterized in that After the 3D CT image is projected using a light projection algorithm to generate a 2D DRR image at a corresponding moment, the method further includes: The 2D DRR image is corrected using a generative adversarial network to obtain a simulated X-ray image C with a real X-ray image style but unchanged image content. proj .

5. The three-dimensional tumor real-time tracking method according to claim 4, It is characterized in that The use of a generative adversarial network to perform image correction on the 2D DRR image is specifically as follows: The generative adversarial network is a CUT network, which adopts contrast learning of multi-level image blocks, takes 2D DRR images as source domains and real X-ray images as target domains, and uses a generator and a discriminator to convert the 2D DRR images into simulated X-ray images with the style of real X-ray images but the image content remains unchanged.

6. The three-dimensional tumor real-time tracking method according to claim 5, It is characterized in that Before inputting the 2D DRR image into the trained tumor tracking network, the method further includes: The simulated X-ray image C proj Input the tumor tracking network, and extract the simulated X-ray image C through the tumor tracking network using the residual block proj The features of the channel layer are used as the third dimension features to output a three-dimensional feature map; The three-dimensional feature map is input into the 3D feature extraction network, which is a registration network based on the combination of Swintransformer and CNN. The 3D feature extraction network outputs a 3D deformation field φ under the joint action of the encoding and decoding mechanism and the self-attention mechanism. p ; The 3D deformation field φ is transformed through the spatial transformation layer p Acting on the reference image M seg , obtain the simulated X-ray image C proj The corresponding tumor prediction segmentation image P seg ; Calculate the tumor prediction segmentation image P seg and floating segmented image S seg The maximum similarity between them is calculated, and the tumor tracking network is iteratively optimized according to the maximum similarity to obtain a trained tumor tracking network.

7. The three-dimensional tumor real-time tracking method according to claim 6, It is characterized in that The tumor tracking network uses the tumor prediction segmentation image P seg and floating segmented image S seg The Dice similarity index and the MSE of the predicted deformation field and the true deformation field, as well as the smoothness constraint of the deformation field are used as the loss function: L=λ 1 L Dice (S seg P seg )+λ 2 L MSE (φ hybrid ,φ p )+λ 3 L Reg (φ p ) In the above formula, n represents the number of categories in the image, k represents the kth category of the image, represents all elements in the entire deformation field, and λ 1 , 2 , 3 Respectively represent L Dice , L MSE , L Reg The weight of .

8. A three-dimensional tumor real-time tracking method and device, It is characterized in that include: Image acquisition module: used to obtain floating images and reference images of the tumor site; Data enhancement module: used for performing mixed data enhancement on the floating image and the reference image to obtain a 3D CT image and a floating segmentation image representing each respiratory phase; Image projection module: used to project the 3D CT image using a light projection algorithm to generate a 2D DRR image at a corresponding moment; Tumor prediction module: used to input the 2D DRR image into the trained tumor tracking network, output the 3D deformation field of the 2D DRR image through the tumor tracking network, apply the 3D deformation field to the reference image, and generate a tumor prediction segmentation image at the corresponding moment of the 2D DRR image.

9. A computer device, It is characterized in that The computer device includes a processor and a memory coupled to the processor, wherein: The memory stores program instructions for implementing the three-dimensional tumor real-time tracking method according to any one of claims 1 to 7; The processor is used to execute the program instructions stored in the memory to control a three-dimensional tumor real-time tracking method.

10. A storage medium, It is characterized in that Program instructions executable by a processor are stored, and the program instructions are used to execute the three-dimensional tumor real-time tracking method according to any one of claims 1 to 7.

Images