Three-dimensional tumor real-time tracking method, device, computer equipment and storage medium

Abstract

This application relates to a method, apparatus, computer device, and storage medium for real-time three-dimensional tumor tracking. The method includes: acquiring a floating image and a reference image of the tumor site; performing hybrid data augmentation on the floating image and the reference image to obtain a 3D CT image and a floating segmentation image representing each respiratory phase; projecting the 3D CT image using a ray projection algorithm to generate a 2D DRR image at the corresponding time; inputting the 2D DRR image into a trained tumor tracking network, outputting a 3D deformation field of the 2D DRR image through the tumor tracking network, and applying the 3D deformation field to the reference image to generate a tumor prediction segmentation image at the corresponding time of the 2D DRR image. This application's embodiment uses single-angle X-ray imaging for tumor tracking, significantly reducing the radiation dose, achieving high-precision tumor tracking at low doses, and significantly improving the speed of tumor tracking.

Description

Technical Field

[0001] This application belongs to the field of medical image processing technology, and specifically relates to a three-dimensional tumor real-time tracking method, device, computer equipment, and storage medium. Background Technology

[0002] The goal of radiotherapy is to eliminate tumor cells to the maximum extent possible while protecting surrounding normal tissues and organs. However, during radiotherapy, the shape and location of organs and tumors may change due to respiratory and cardiac movements, affecting the accuracy of radiotherapy. To ensure that the irradiation field closely follows the target area while avoiding damage to surrounding normal tissues and organs, it is necessary to rapidly acquire images during radiotherapy and register them with images used to develop the radiotherapy plan. This allows for the real-time location of the tumor during radiotherapy and enables real-time tracking of tumor cells.

[0003] Existing tumor tracking methods typically include the following two types:

[0004] 1. Tracking using markers: This method includes internal and external markers. Internal markers are directly implanted into the tumor or surrounding tissue, allowing for real-time acquisition of tumor location information through continuous imaging, offering high accuracy. However, this method is invasive and prone to problems such as pneumothorax infection and immune rejection. External markers use an external substitute as a replacement for the internal marker. This method requires establishing a relationship model between the external substitute and the tumor, but this model may change as the respiratory search and radiotherapy process progresses, leading to inaccurate localization and severely affecting the precision of radiotherapy.

[0005] 2. Tumor localization using image registration: This method acquires the mapping deformation field between 2D and 3D images to automatically propagate the tumor contour from pre-radiotherapy images to images during radiotherapy. This method can be divided into model-based registration and data-driven registration. Model-based registration attempts to utilize dimensionality reduction models based on B-Spline or PCA to extract principal coefficients or components, establish a patient space based on specific priors, and complete the lung search model through iterative optimization. However, this method typically requires iterative optimization of each image to be registered, resulting in slow registration speed and failing to meet the real-time requirements of tumor localization. With a large amount of data for model training, data-driven registration can be directly applied in the testing phase without retraining, thus significantly shortening registration time. However, the 2D images used in this method usually require multiple X-ray or CBCT projections taken from multiple angles, resulting in a large radiation dose, potentially causing additional harm to the patient, and a long registration time. Summary of the Invention

[0006] This application provides a three-dimensional tumor real-time tracking method, apparatus, computer device, and storage medium, which aims to at least partially solve one of the aforementioned technical problems in the prior art.

[0007] To address the above problems, this application provides the following technical solution:

[0008] A three-dimensional tumor real-time tracking method, comprising:

[0009] Acquire floating images and reference images of the tumor site;

[0010] Hybrid data augmentation is performed on the floating image and the reference image to obtain 3D CT images and floating segmented images representing each respiratory phase;

[0011] The 3D CT image is projected using a ray projection algorithm to generate a 2D DRR image at the corresponding time point;

[0012] The 2D DRR image is input into a trained tumor tracking network, which outputs a 3D deformation field of the 2D DRR image. The 3D deformation field is then applied to a reference image to generate a tumor prediction segmentation image of the 2D DRR image at the corresponding time.

[0013] The technical solution adopted in this application embodiment also includes: the specific method of utilizing the floating image and reference image of the tumor site is as follows:

[0014] The end of exhalation is used as a floating image M. CT The remaining respiratory phases are used as reference images M. seg .

[0015] The technical solution adopted in this application embodiment further includes: the specific steps of performing hybrid data enhancement on the floating image and the reference image to obtain 3D CT images and floating segmented images representing each respiratory phase are as follows:

[0016] The floating image M was registered using the Demons image registration algorithm. CT and reference image M seg Registration was performed sequentially to obtain the deformation φ between each breathing phase. 1,...i,j ;

[0017] From the deformation φ 1,...i,j Two random samples are randomly selected and their weights are superimposed to obtain the phase deformation from the end of exhalation to the other respiratory phases;

[0018] By using a thin-plate spline interpolation algorithm to simulate random control points and random movement distances, the in-phase deformation of a simulated breathing phase is obtained.

[0019] Based on the interphase deformation and intraphase deformation, a mixed deformation field φ is obtained. ybrid and the hybrid deformation field φ ybrid Applying the CT to the end of expiration, 3D CT images representing each expiratory phase are obtained. CT .

[0020] The technical solution adopted in this application embodiment further includes: after projecting the 3D CT image using a ray projection algorithm to generate a 2D DRR image at the corresponding time, it further includes:

[0021] Generative adversarial networks are used to perform image correction on the 2D DRR image to obtain a simulated X-ray image C that retains the style of a real X-ray image but whose content remains unchanged. proj .

[0022] The technical solution adopted in this application embodiment further includes: the image correction of the 2D DRR image using a generative adversarial network specifically involves:

[0023] The generative adversarial network is a CUT network. The CUT network uses multi-level image patch contrastive learning, takes 2DDRR images as the source domain and real X-ray images as the target domain, and uses a generator and a discriminator to convert the 2DDRR images into simulated X-ray images that have the style of real X-ray images but whose image content remains unchanged.

[0024] The technical solution adopted in this application embodiment further includes: before inputting the 2D DRR image into the trained tumor tracking network, it further includes:

[0025] The simulated X-ray image C proj Input to the tumor tracking network, and extract the simulated X-ray image C using residual blocks through the tumor tracking network. proj The features are analyzed, and the channel layer is used as the third dimension of the feature to output a three-dimensional feature map.

[0026] The three-dimensional feature map is input into a 3D feature extraction network, which is a registration network based on a combination of Swintransformer and CNN. Under the combined action of encoding / decoding mechanism and self-attention mechanism, the 3D feature extraction network outputs a 3D deformation field φ. p ;

[0027] The 3D deformation field φ is transformed through a spatial transformation layer. p Acting on reference image M seg The simulated X-ray image C was obtained. proj The corresponding tumor prediction segmentation image P seg ;

[0028] Calculate the tumor prediction segmentation image P seg and floating segmented image S seg The maximum similarity between the two is used to iteratively optimize the tumor tracking network, resulting in a trained tumor tracking network.

[0029] The technical solution adopted in this application embodiment also includes: the tumor tracking network uses a tumor prediction segmentation image P seg and floating segmented image S seg The loss function is composed of the Dice similarity index between the predicted and actual deformation fields, the MSE of the deformation field and the smoothing constraint of the deformation field.

[0030]

[0031]

[0032] L=λ1L Dice (S seg P seg )+λ2L MSE (φ hybrid , φ p )+λ3L Reg (φ p )

[0033] In the above formula, n represents the number of categories in the image, k represents the k-th category of the image, λ1 represents all elements in the entire deformation field, and λ2, λ3 represent L1, L2, and L3 respectively. Dice L MSE L Reg The weight.

[0034] Another technical solution adopted in this application embodiment is: a three-dimensional tumor real-time tracking method device, comprising:

[0035] Image acquisition module: used to acquire floating images and reference images of the tumor site;

[0036] Data augmentation module: used to perform hybrid data augmentation on the floating image and the reference image to obtain 3D CT images and floating segmented images representing each respiratory phase;

[0037] Image projection module: used to project the 3D CT image using a light projection algorithm to generate a 2D DRR image at the corresponding time.

[0038] Tumor prediction module: It is used to input the 2D DRR image into the trained tumor tracking network, output the 3D deformation field of the 2D DRR image through the tumor tracking network, apply the 3D deformation field to the reference image, and generate the tumor prediction segmentation image of the 2D DRR image at the corresponding time.

[0039] Another technical solution adopted in this application embodiment is: a computer device, the computer device including a processor and a memory coupled to the processor, wherein,

[0040] The memory stores program instructions for implementing the three-dimensional tumor real-time tracking method;

[0041] The processor is used to execute the program instructions stored in the memory to control the three-dimensional tumor real-time tracking method.

[0042] Another technical solution adopted in this application embodiment is: a storage medium storing processor-executable program instructions, the program instructions being used to execute the three-dimensional tumor real-time tracking method.

[0043] Compared to existing technologies, the beneficial effects of the embodiments of this application are as follows: The three-dimensional tumor real-time tracking method, apparatus, computer equipment, and storage medium of the embodiments of this application obtain 3D CT images and floating segmented images representing various respiratory phases of the lungs by performing mixed data augmentation on floating images and reference images. The 3D CT images are then projected using a ray projection algorithm to generate 2DDRR images at corresponding times. These 2DDRR images are input into a trained tumor tracking network, which utilizes a network framework combining Swing Transformer and CNN to achieve tumor tracking based on a single X-ray image. The embodiments of this application perform tumor tracking based on single-angle X-ray images, greatly reducing the radiation dose and enabling low-dose, high-precision 3D tumor tracking in a shorter time. This is beneficial for image-guided surgery and radiotherapy, while simplifying the hardware of tomographic imaging systems. Furthermore, the embodiments of this application utilize a combination of Swing Transformer and CNN for tumor tracking, significantly improving the speed of tumor tracking while retaining the self-attention mechanism to achieve higher accuracy. Attached Figure Description

[0044] Figure 1 This is a flowchart of a three-dimensional tumor real-time tracking method according to an embodiment of this application;

[0045] Figure 2 This application embodiment relates to the floating image M. CT and reference image S seg A flowchart for performing hybrid data augmentation;

[0046] Figure 3 This is a schematic diagram of the image correction framework according to an embodiment of this application;

[0047] Figure 4 This is a schematic diagram of the tumor tracking network structure according to an embodiment of this application;

[0048] Figure 5 This is a schematic diagram of a tumor registration experiment performed on patient data with tumors, from the end of expiration to the end of inspiration.

[0049] Figure 6 A schematic diagram showing the experimental results for three types of tumors: large, medium, and small.

[0050] Figure 7 This is a schematic diagram illustrating the results of incorporating real CBCT projections into a tumor tracking network for tumor localization.

[0051] Figure 8 This is a schematic diagram of the structure of the three-dimensional tumor real-time tracking method device according to an embodiment of this application;

[0052] Figure 9 This is a schematic diagram of the computer device structure according to an embodiment of this application;

[0053] Figure 10 This is a schematic diagram of the structure of the storage medium according to an embodiment of this application. Detailed Implementation

[0054] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0055] The terms "first," "second," and "third" in this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. All directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of this application are only used to explain the relative positional relationships and movements between components in a specific orientation (as shown in the figures). If the specific orientation changes, the directional indications also change accordingly. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or computer device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or computer devices.

[0056] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0057] Please see Figure 1 This is a flowchart of a three-dimensional tumor real-time tracking method according to an embodiment of this application. The three-dimensional tumor real-time tracking method according to an embodiment of this application includes the following steps:

[0058] S100: Acquire floating images and reference images of the tumor site;

[0059] In this step, medical image registration involves performing a geometric transformation on one image and mapping it to another, ensuring that relevant points in the two images are spatially consistent. The image to which the geometric transformation is applied is defined as the moving image, and the other stationary image is defined as the fixed image. In this embodiment, a 4DCT image of the lungs is selected as the experimental subject, and the end of expiration is used as the moving image M. CT The remaining respiratory phases are used as reference images M. seg Specifically, it can also be CT images of other parts of the body such as the heart.

[0060] S110: For floating image M CT and reference image M seg Hybrid data augmentation was performed to obtain 3DCT images S representing each respiratory phase. CT and floating segmented image S seg ;

[0061] In this step, since real, simultaneous 2D / 3D medical images are extremely scarce, this embodiment of the application uses data augmentation to obtain a large number of 3D CT images representing various respiratory phases of the lungs. Specifically, as... Figure 2 As shown, this application embodiment uses a floating image M. CT and reference image M seg The flowchart for hybrid data augmentation includes the following steps:

[0062] S111: Using the Demons image registration algorithm to register the floating image M CT and reference image M seg Registration was performed sequentially to obtain the deformation φ between each breathing phase. 1,...i,j ;

[0063] S112: From deformation v1,...i,j Two random samples are randomly selected and weighted to obtain a large number of interphase deformations from the end of exhalation to the other respiratory phases;

[0064] S113: Using the thin plate spline interpolation algorithm, random control points and random movement distances are simulated to obtain a large number of intraphase deformations simulating a certain breathing phase;

[0065] S114: A large amount of mixed deformation field φ is obtained based on inter-phase deformation and intra-phase deformation. ybrid and the mixed deformation field φ ybrid Applying the CT to the end of expiration, 3D CT images representing each expiratory phase of the lungs are obtained. CT .

[0066] S120: Using a ray projection algorithm to process 3D CT images S CT Project the image to generate a 2DDRR (Digitally Reconstructed Radiograph) image at the corresponding time.

[0067] In this step, a ray projection algorithm is used to image the 3D CT image S. CT Projection is performed to generate 2D digital reconstructed images at the corresponding time points to simulate real X-ray images.

[0068] S130: Image correction of 2D DRR images is performed using a generative adversarial network to obtain simulated X-ray images C that retain the style of realistic X-ray images while preserving the image content. proj ;

[0069] In this step, to achieve 2D / 3D image registration, the data-enhanced 3DCT image (SCT) needs to be projected onto the 2D image using light projection to simulate a real X-ray image. However, the 2DDRR still differs from the real X-ray image in terms of noise and grayscale. To address this issue, this embodiment uses GAN (Generative Adversarial Networks) to correct the 2DDRR image, ensuring that the corrected 2DDRR retains the original image content while possessing the style of a real X-ray image. Specifically, this embodiment selects a CUT network (Contrastive Learning for Unpaired Image-to-Image Translation), employing multi-level image patch contrastive learning, and using a generator and a discriminator to achieve unidirectional image correction. For details... Figure 3The diagram shown is a schematic of the image correction framework of an embodiment of this application. In this embodiment, DRR is a simulated two-dimensional image and X-ray is a real X-ray image. DRR is used as the source domain and X-ray is used as the target domain. With the resource-saving approach, a generator and a discriminator are used to convert the DRR image into a simulated X-ray image with the style of a real X-ray image.

[0070] S140: Simulate X-ray image C proj Input to the tumor tracking network, and output a 3D deformation field φ through the tumor tracking network. p Applying a 3D deformation field φp to the reference image M seg Generate simulated X-ray images C proj The corresponding tumor prediction segmentation image P seg And use tumor prediction segmentation image P seg and floating segmented image S seg The maximum similarity between the tumor tracking networks is used to iteratively optimize the tumor tracking network, resulting in a well-trained tumor tracking network.

[0071] In this step, the tumor tracking network utilizes the combined capabilities of the Swing Transformer network and the CNN network to achieve tumor tracking based on a single X-ray image. Specifically, as shown... Figure 4 The diagram shown is a schematic of the tumor tracking network structure according to an embodiment of this application. The tumor tracking network employs a dimensionality-upgrading method, treating the channel layer as a third-dimensional feature, transforming the 2D / 3D registration problem into a 3D / 3D registration problem. First, it uses residual blocks to extract the simulated X-ray image C. proj The features are analyzed, and the channel layer is used as the third dimension of the feature map to output a three-dimensional feature map. This three-dimensional feature map is then input into a 3D feature extraction network, which is a registration network based on a combination of Swintransformer and CNN. Under the combined action of the encoding / decoding mechanism and the self-attention mechanism, it outputs a 3D deformation field φ. p Then, the 3D deformation field φ is transformed through a spatial transformation layer. p Acting on reference image M seg To obtain simulated X-ray images C proj The corresponding tumor prediction segmentation image P seg And by calculating the tumor prediction segmentation image P seg and floating segmented image S seg The maximum similarity between the tumor tracking network and the corresponding network is used to iteratively optimize the network and complete its training. This embodiment utilizes a combination of Swing Transformer and CNN for tumor tracking, achieving higher accuracy while retaining the self-attention mechanism, and also accelerating the tumor tracking speed.

[0072] Furthermore, during model training, this embodiment uses the Dice similarity index between the predicted tumor segmentation image and the actual floating segmentation image, the MSE (Mean Square Error) of the predicted deformation field and the actual deformation field, and the smoothing constraint of the deformation field as loss functions:

[0073]

[0074]

[0075] L=λ1L Dice (S seg P seg )+λ2L MSE (φ hybrid , φ p )+λ3L Reg (φ p (3)

[0076] In the above formula, n represents the number of categories in the image, k represents the k-th category of the image, and λ represents all elements in the entire deformation field. λ1, λ2, and λ3 represent L, L, and L, respectively. Dice L MSE L Reg The weights are 1.0, 0.1, and 0.01 respectively in this embodiment.

[0077] S150: Input a single 2D X-ray image or 2D DRR image into a trained tumor tracking network, output a predicted 3D deformation field through the tumor tracking network, and apply the 3D deformation field to a reference image to obtain a 3D tumor image at the corresponding time of the single 2D X-ray image or 2D DRR image.

[0078] To verify the feasibility and effectiveness of the embodiments of this application, the following embodiments conducted a tumor registration experiment on patient data with tumors, from the end of expiration to the end of inspiration. The experimental results are as follows: Figure 5 As shown, odd-numbered columns represent unregistered images, and even-numbered rows represent registered images. Specifically, the first and second rows show the shape of the lungs before and after registration (segmented images), while the third and fourth rows show the images of the tumors before and after registration. Experiments were also conducted on dynamic lung phantoms with three different tumor sizes, from the end of expiration to the end of inspiration. Specific experimental results are shown below. Figure 6 As shown, from left to right, the registration results for three types of tumors—large, medium, and small—are displayed. Odd-numbered columns represent unregistered images, while even-numbered rows represent registered images. It can be seen that good tumor tracking results were obtained for both real patients and phantom experiments.

[0079] In addition, we evaluated the phantom using real X-ray images, specifically as follows: Figure 7 The diagram illustrates the results of incorporating real CBCT projections into a tumor tracking network for tumor localization. The left side (a) shows the result with CUT correction, and the right side (b) shows the result without CUT correction. From top to bottom, the target area delineation curves for large, medium, and small tumors are shown. The black outline represents the gold standard, the white outline represents the predicted value, and the crosses within the outlines represent the reference tumor centroid and the predicted tumor centroid, respectively. Ablation experiments with and without image correction demonstrate that the tumor image prediction results obtained using the tumor tracking network proposed in this application are very close to the tumor in the real CBCT projection drawn by the physician (the root mean square error of the tumor centroid is less than 1.5 mm), achieving high tracking accuracy.

[0080] Based on the above, the three-dimensional tumor real-time tracking method of this application embodiment obtains 3D CT images and floating segmented images representing various respiratory phases of the lungs by performing hybrid data augmentation on floating images and reference images. The 3D CT images are then projected using a ray projection algorithm to generate 2DDRR images at corresponding times. The reconstructed images are corrected using a CUT network to obtain simulated X-ray images that retain the style of real X-ray images while maintaining the original image content. These simulated images are then input into a tumor tracking network, which uses a network framework combining Swing Transformer and CNN to achieve tumor tracking based on a single X-ray image. This application embodiment performs tumor tracking based on single-angle X-ray images, significantly reducing the radiation dose and enabling low-dose, high-precision 3D tumor tracking in a shorter time. This is beneficial for image-guided surgery and radiotherapy, while also simplifying the hardware of tomographic imaging systems. Furthermore, this application embodiment utilizes a combination of Swing Transformer and CNN for tumor tracking, significantly improving the speed of tumor tracking while retaining the self-attention mechanism for higher accuracy.

[0081] Please see Figure 8 This is a schematic diagram of the structure of a three-dimensional tumor real-time tracking method device according to an embodiment of this application. The three-dimensional tumor real-time tracking method device 40 according to an embodiment of this application includes:

[0082] Image acquisition module 41: used to acquire floating images and reference images of the tumor site;

[0083] Data augmentation module 42: used to perform hybrid data augmentation on the floating image and the reference image to obtain 3D CT images and floating segmented images representing each respiratory phase;

[0084] Image projection module 43: used to project the 3D CT image using a light projection algorithm to generate a 2D DRR image at the corresponding time.

[0085] Tumor prediction module 44: It is used to input the 2D DRR image into the trained tumor tracking network, output the 3D deformation field of the 2D DRR image through the tumor tracking network, apply the 3D deformation field to the reference image, and generate the tumor prediction segmentation image of the 2D DRR image at the corresponding time.

[0086] Please see Figure 9 This is a schematic diagram of a computer device structure according to an embodiment of this application. The computer device 50 includes:

[0087] Memory 51 storing executable program instructions;

[0088] Processor 52 connected to memory 51;

[0089] The processor 52 is used to call the executable program instructions stored in the memory 51 and perform the following steps: acquire a floating image and a reference image of the tumor site; perform hybrid data augmentation on the floating image and the reference image to obtain a 3D CT image and a floating segmentation image representing each respiratory phase; project the 3D CT image using a ray projection algorithm to generate a 2D DRR image at the corresponding time; input the 2D DRR image into a trained tumor tracking network, output the 3D deformation field of the 2D DRR image through the tumor tracking network, and apply the 3D deformation field to the reference image to generate a tumor prediction segmentation image at the corresponding time of the 2D DRR image.

[0090] The processor 52 can also be referred to as a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip with signal processing capabilities. The processor 52 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor.

[0091] Please see Figure 10 , Figure 10This is a schematic diagram of the structure of the storage medium in an embodiment of this application. The storage medium in this embodiment stores program instructions 61 capable of implementing the following steps: acquiring a floating image and a reference image of the tumor site; performing hybrid data augmentation on the floating image and the reference image to obtain a 3D CT image and a floating segmentation image representing each respiratory phase; projecting the 3D CT image using a ray projection algorithm to generate a 2D DRR image at the corresponding time; inputting the 2D DRR image into a trained tumor tracking network, outputting a 3D deformation field of the 2D DRR image through the tumor tracking network, and applying the 3D deformation field to the reference image to generate a tumor prediction segmentation image at the corresponding time of the 2D DRR image. The program instructions 61 can be stored in the aforementioned storage medium in the form of a software product, including several instructions to cause a computer device (which may be a personal computer, server, or network computer device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage media include: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program instructions, or terminal computer devices such as computers, servers, mobile phones, and tablets. Servers can be independent servers or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0092] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, apparatuses, or units, and may be electrical, mechanical, or other forms.

[0093] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for real-time three-dimensional tumor tracking, characterized in that, include: Acquire floating images and reference images of the tumor site; Hybrid data augmentation is performed on the floating image and the reference image to obtain 3D CT images and floating segmented images representing each respiratory phase; The 3D CT image is projected using a ray projection algorithm to generate a 2D DRR image at the corresponding time point; The 2D DRR image is input into a trained tumor tracking network, which outputs a 3D deformation field from the 2D DRR image. This 3D deformation field is then applied to a reference image to generate a tumor prediction and segmentation image corresponding to the given time in the 2D DRR image. The specific steps of performing hybrid data augmentation on the floating image and the reference image to obtain 3D CT images and floating segmented images representing each respiratory phase are as follows: The floating image was registered using the Demons image registration algorithm. M CT and reference image M seg Registration was performed sequentially to obtain the deformation between each breathing phase. 1,...i,j ; From the deformation 1,...i,j Two random samples are randomly selected and their weights are superimposed to obtain the phase deformation from the end of exhalation to the other respiratory phases; By using a thin-plate spline interpolation algorithm to simulate random control points and random movement distances, the in-phase deformation of a simulated breathing phase is obtained. A mixed deformation field is obtained based on the interphase deformation and intraphase deformation. and the hybrid deformation field Applying the CT scan to the end of expiration, 3D CT images representing each phase of expiration are obtained. S CT .

2. The three-dimensional tumor real-time tracking method according to claim 1, characterized in that, The acquisition of the floating image and reference image of the tumor site specifically involves: The end of exhalation is used as a floating image. M CT The remaining respiratory phases were used as reference images. M seg .

3. The three-dimensional tumor real-time tracking method according to any one of claims 1 to 2, characterized in that, After projecting the 3D CT image using a ray projection algorithm to generate a 2D DRR image at the corresponding time, the method further includes: Generative adversarial networks are used to perform image correction on the 2D DRR images to obtain simulated X-ray images that retain the style of realistic X-ray images while preserving their content. C proj .

4. The three-dimensional tumor real-time tracking method according to claim 3, characterized in that, The specific steps of using a generative adversarial network to perform image correction on the 2D DRR image are as follows: The generative adversarial network is a CUT network. The CUT network uses multi-level image patch contrastive learning, takes 2D DRR images as the source domain and real X-ray images as the target domain, and uses a generator and a discriminator to convert the 2D DRR images into simulated X-ray images with the style of real X-ray images but with unchanged image content.

5. The three-dimensional tumor real-time tracking method according to claim 4, characterized in that, Before inputting the 2D DRR image into the trained tumor tracking network, the method further includes: The simulated X-ray image C proj The simulated X-ray image is input into a tumor tracking network, and extracted using residual blocks through the tumor tracking network. C proj The features are analyzed, and the channel layer is used as the third dimension of the feature to output a three-dimensional feature map. The 3D feature map is input into a 3D feature extraction network, which is a registration network based on a combination of Swintransformer and CNN. Under the combined action of encoding / decoding mechanism and self-attention mechanism, the 3D feature extraction network outputs a 3D deformation field. ; The 3D deformation field is transformed through a spatial transformation layer. Acting on reference image M seg To obtain the simulated X-ray image C proj Corresponding tumor prediction segmentation image P seg ; Calculate the tumor prediction segmentation image P seg and floating segmented images S seg The maximum similarity between the two is used to iteratively optimize the tumor tracking network, resulting in a trained tumor tracking network.

6. The three-dimensional tumor real-time tracking method according to claim 5, characterized in that, The tumor tracking network uses tumor prediction and segmentation images. P seg and floating segmented images S seg The loss function is composed of the Dice similarity index between the predicted and actual deformation fields, the MSE of the deformation field and the smoothing constraint of the deformation field. In the above formula, Indicates the number of categories in the image. k The image represents the first k Categories This represents all elements in the entire deformation field. , , They represent , , The weight.

7. A three-dimensional tumor real-time tracking method apparatus utilizing the three-dimensional tumor real-time tracking method of claim 1, characterized in that, include: Image acquisition module: used to acquire floating images and reference images of the tumor site; Data augmentation module: used to perform hybrid data augmentation on the floating image and the reference image to obtain 3D CT images and floating segmented images representing each respiratory phase; Image projection module: used to project the 3D CT image using a light projection algorithm to generate a 2DDRR image at the corresponding time. Tumor prediction module: It is used to input the 2D DRR image into the trained tumor tracking network, output the 3D deformation field of the 2D DRR image through the tumor tracking network, apply the 3D deformation field to the reference image, and generate the tumor prediction segmentation image of the 2D DRR image at the corresponding time.

8. A computer device, characterized in that, The computer device includes a processor and a memory coupled to the processor, wherein, The memory stores program instructions for implementing the three-dimensional tumor real-time tracking method according to any one of claims 1-6; The processor is used to execute the program instructions stored in the memory to control the three-dimensional tumor real-time tracking method.

9. A storage medium, characterized in that, The device stores processor-executable program instructions for performing the three-dimensional tumor real-time tracking method according to any one of claims 1 to 6.

Images