Image registration method and device for tumor tracking, and network model establishment method

Abstract

The present application provides an image registration method and device for tumor tracking, and a network model establishment method. The image registration method for tumor tracking comprises: acquiring an end-of-exhalation image as a first floating image; acquiring two orthogonal angle-based 2D images of a target phase; and inputting the first floating image and the two orthogonal angle-based 2D images into a pre-trained registration network model to obtain a 3D registration image of tumor tracking during radiotherapy. By means of the solution, the radiation dose can be effectively reduced, and the real-time requirement can be met while ensuring high precision.

Description

Image registration method and device for tumor tracking, and network model establishment method Technical Field

[0001] The present application belongs to the field of medical device technology, and in particular relates to an image registration method and device for tumor tracking, and a network model establishment method. Background Art

[0002] The goal of radiotherapy is to kill tumor cells as much as possible while protecting normal tissue. However, the shape and position of organs and tumors will change due to respiratory movement, cardiac movement and the progress of radiotherapy. If you want to ensure that the irradiation field closely follows the target area without harming the surrounding normal tissues and organs, you need to collect images during radiotherapy and align them with the images used to formulate the radiotherapy plan. For images used to formulate radiotherapy plans, 3D images have rich anatomical structure information and are a better choice for formulating radiotherapy plans. However, for images used during radiotherapy, the reconstruction of 3D images takes a certain amount of time, and the subsequent alignment with the pre-radiotherapy images takes even more time, which makes it difficult to meet the needs of real-time radiotherapy.

[0003] In order to ensure real-time requirements, considering that 2D images have a faster imaging speed, 2D images will also be used in radiotherapy images. 2D / 3D registration can be divided into traditional image registration and deep learning-based image registration. For traditional image registration, it still needs to be iteratively optimized one by one during the testing phase. The registration speed is slow, usually taking several minutes or even dozens of minutes, which is difficult to meet the real-time requirements in radiotherapy. With the development of artificial intelligence and deep learning, the registration method based on deep learning replaces the tedious iterative optimization process in the testing process with predicted values, which can improve computational efficiency to a certain extent. However, the existing deep learning-based registration method has a large dose (that is, 2D images are usually multiple and multi-angle), which leads to the problem of too long registration time.

[0004] Currently, no effective solution has been proposed to the problem that the existing 2D / 3D registration takes too long and cannot meet the real-time requirements during radiotherapy.

[0005] Summary of the Invention

[0006] The purpose of this application is to provide an image registration method and device, and a network model establishment method for tumor tracking, which can achieve the technical effect of reducing radiation dose and achieving high-precision registration while meeting real-time requirements.

[0007] The present application provides an image registration method and apparatus for tumor tracking, and a network model establishment method, which are implemented as follows:

[0008] An image registration method for tumor tracking, the method comprising:

[0009] acquiring an end-of-respiration image as a first floating image;

[0010] Acquire two 2D images of the target phase based on orthogonal angles;

[0011] The first floating image and the two 2D images based on orthogonal angles are input into the registration network model to obtain a 3D registered image for tumor tracking during radiotherapy.

[0012] In one embodiment, the registration network model is established as follows:

[0013] acquiring a respiratory image of a predetermined phase in the 4D CT image as a second floating image;

[0014] performing data expansion based on the second floating image to obtain 3D CT images of each expiratory phase;

[0015] By projecting light, the obtained 3D CT images of each expiratory phase are projected to obtain two 2D DRRs at orthogonal angles corresponding to each respiratory phase;

[0016] The registration network model is obtained by performing model training on the two 2D DRRs at orthogonal angles corresponding to each respiratory phase and the second floating image.

[0017] In one embodiment, the registration network model is obtained by performing model training on the two 2D DRRs at orthogonal angles corresponding to each respiratory phase and the second floating image, including:

[0018] The residual block is used to extract the features of 2D DRR, and the channel layer is used as the third dimension feature to output a three-dimensional feature map;

[0019] Inputting the three-dimensional feature map and the second floating image into a 3D feature extraction network to obtain a three-dimensional deformation field;

[0020] Applying the three-dimensional deformation field to the second floating image through a spatial transformation layer to obtain a predicted 3D CT image corresponding to the 2D DRR;

[0021] The maximum similarity between the predicted 3D CT image and the real 3D CT image obtained by data expansion is calculated to iteratively optimize the network model and train the registration network model.

[0022] In one embodiment, in calculating the maximum similarity between the predicted 3D CT image and the true 3D CT image obtained by data augmentation to iteratively optimize the network model and train the registration network model, the following formula is used as the loss function: L=λ1L Dice (F seg ,P seg )+λ2L MI (F CT ,P CT )+λ3L Reg (φ p )

[0023] Among them, F represents the true value, P represents the predicted value, n represents the number of categories in the image, i represents the i-th category of the image, p represents all elements in the entire deformation field, λ1, λ2, and λ3 represent L Dice , L MI , L Reg The weight, L Dice represents the loss between two segmentation labels, L MI Denotes the loss between two images, L Reg represents the regularization loss of the deformation field, F seg represents the segmentation label of the real 3D CT image, P seg Denotes the predicted segmentation label of the 3D CT image, F CT represents the real 3D CT image, P CT represents the predicted 3D CT image, φ p represents the predicted deformation field.

[0024] In one embodiment, data expansion is performed based on the second floating image to obtain 3D CT images of each expiratory phase, including:

[0025] obtaining a deformation field between a plurality of phases according to the second floating image;

[0026] Multiple mixed deformation fields are obtained by superimposing the deformation fields within a phase on the deformation fields between multiple phases;

[0027] A plurality of mixed deformation fields are applied to the end-of-exhalation image to obtain a 3D CT image of each exhalation phase.

[0028] In one embodiment, obtaining a deformation field between a plurality of phases according to the second floating image includes:

[0029] Matching the other M phases through the second floating image to obtain the deformation field between the M phases, where M is 7 or 9;

[0030] Two of the M inter-phase deformation fields are randomly selected and superimposed with random weights to obtain multiple inter-phase deformation fields from the end of exhalation to the remaining phases.

[0031] In one embodiment, the phase deformation field is obtained as follows:

[0032] Set the random control point and random movement distance of the target phase;

[0033] According to the set random control points and random moving distances, thin plate spline interpolation simulation is used to obtain the deformation field in multiple phases of the target phase.

[0034] In one embodiment, the two 2D images based on orthogonal angles include: 2D X-ray projections and / or 2D DRR.

[0035] A registration network model establishment method, comprising:

[0036] Acquire the respiratory end image in the 4D CT image as a floating image;

[0037] Performing data expansion based on the floating image to obtain 3D CT images of each expiratory phase;

[0038] By projecting light, the obtained 3D CT images of each expiratory phase are projected to obtain two 2D DRRs at orthogonal angles corresponding to each respiratory phase;

[0039] The registration network model is obtained by performing model training on the two 2D DRRs at orthogonal angles corresponding to each respiratory phase and the floating image.

[0040] In one embodiment, the registration network model is obtained by performing model training on the two 2D DRRs at orthogonal angles corresponding to each respiratory phase and the floating image, including:

[0041] The residual block is used to extract the features of 2D DRR, and the channel layer is used as the third dimension feature to output a three-dimensional feature map;

[0042] Inputting the three-dimensional feature map and the floating image into a 3D feature extraction network to obtain a three-dimensional deformation field;

[0043] Applying the three-dimensional deformation field to the floating image through a spatial transformation layer to obtain a predicted 3D CT image corresponding to the 2D DRR;

[0044] The maximum similarity between the predicted 3D CT image and the real 3D CT image obtained by data expansion is calculated to iteratively optimize the network model and train the registration network model.

[0045] In one embodiment, data expansion is performed based on the floating image to obtain 3D CT images of each expiratory phase, including:

[0046] obtaining a plurality of phase-to-phase deformation fields according to the floating image;

[0047] Multiple mixed deformation fields are obtained by superimposing the deformation fields within a phase on the deformation fields between multiple phases;

[0048] A plurality of mixed deformation fields are applied to the end-of-exhalation image to obtain a 3D CT image of each exhalation phase.

[0049] An image registration device for tumor tracking, comprising:

[0050] A first acquisition module is used to acquire an end-of-respiration image as a first floating image;

[0051] The second acquisition module is used to acquire two 2D images of the target phase based on orthogonal angles;

[0052] The registration module is used to input the first floating image and the two 2D images based on orthogonal angles into the registration network model to obtain a 3D registered image for tumor tracking during radiotherapy.

[0053] A terminal device comprises a processor and a memory for storing instructions executable by the processor, wherein the steps of the above method are implemented when the processor executes the instructions.

[0054] A computer-readable storage medium stores a computer program / instruction thereon, which implements the steps of the above method when executed by a processor.

[0055] This application provides an image registration method for tumor tracking. By acquiring a respiratory endpoint image and two orthogonal 2D images, and then inputting these images into a registration network model, a 3D registered image is generated for tumor tracking during radiotherapy. In the above example, image registration is achieved by selecting only two orthogonal projections, effectively reducing radiation dose and meeting real-time requirements while ensuring high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0057] FIG1 is a flowchart of an embodiment of an image registration method for tumor tracking provided by the present application;

[0058] FIG2 is a flow chart of an embodiment of a method for establishing a registration network model provided by the present application;

[0059] FIG3 is a logic diagram of an embodiment of a 2D / 3D non-rigid image registration method for lung tumor tracking provided by the present application;

[0060] FIG4 is a hardware structure block diagram of an electronic device for an image registration method for tumor tracking provided by the present application;

[0061] FIG5 is a schematic diagram of the module structure of an embodiment of an image registration device for tumor tracking provided by the present application. DETAILED DESCRIPTION

[0062] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0063] FIG1 is a method flow chart of an embodiment of an image registration method for tumor tracking provided by the present application. Although the present application provides method operation steps or device structures as shown in the following embodiments or drawings, more or fewer operation steps or module units may be included in the method or device based on routine or no creative labor. In the steps or structures where there is no necessary causal relationship logically, the execution order of these steps or the module structure of the device is not limited to the execution order or module structure described in the embodiments of the present application and shown in the drawings. When the method or module structure is applied to an actual device or terminal product, it can be connected according to the method or module structure shown in the embodiment or drawings for sequential execution or parallel execution (for example, a parallel processor or multi-threaded processing environment, or even a distributed processing environment).

[0064] Specifically, as shown in FIG1 , the above-mentioned image registration method for tumor tracking may include the following steps:

[0065] Step 101: Acquire an end-of-respiration image as a first floating image;

[0066] Step 102: Acquire two 2D images of the target phase based on orthogonal angles;

[0067] Step 103: Input the first floating image and the two 2D images based on orthogonal angles into the registration network model to obtain a 3D registered image for tumor tracking during radiotherapy.

[0068] The two orthogonal 2D images may include 2D X-ray projections or 2D DRRs. Specifically, during implementation, a respiratory end image and two orthogonal 2D images are input into a registration network model to obtain a registration result.

[0069] In the above example, by acquiring a respiratory endpoint image and two orthogonal 2D images, and then inputting these into a registration network model, a 3D registered image for tumor tracking during radiotherapy is generated. In this example, image registration is achieved using only two orthogonal projections, effectively reducing radiation dose and achieving high accuracy to meet real-time requirements.

[0070] For 2D / 3D registration, spatial dimensional consistency must be considered first. In this example, a dimensionality increase method is used to transform the 2D / 3D registration problem into a 3D / 3D registration problem. First, residual blocks are used to extract features from the 2D DRR, and the channel layer is used as the third dimension of features to output a 3D feature map. This 3D feature map and the floating image (i.e., the end-expiratory MCT image after data augmentation) are then input into a 3D feature extraction network. The 3D feature extraction network is a 3D / 3D registration network based on the attention mechanism. Through the combined effects of the encoding, decoding, and attention mechanisms, the network outputs a 3D deformation field. This 3D deformation field is then applied to the floating image via a spatial transformation layer to obtain the corresponding 3D CT image PCT corresponding to the predicted DRR (PCT is the model output image and the predicted 3D CT corresponding to the 2D DRR, while FCT is the predicted value). The ground truth (FCT is the true 3D CT obtained from the mixed data augmentation in S1 and is the ground truth) is known. Model training is completed by calculating the maximum similarity between the predicted and ground truth values ​​and continuously optimizing through iterative optimization.

[0071] Based on this, a method for establishing a registration network model is provided in this example, as shown in Figure 2, including the following steps:

[0072] Step 201: Acquire a respiratory image of a predetermined phase in a 4D CT (Computed Tomography) image as a second floating image; that is, an image of a certain phase in the 4D CT image is used as the floating image;

[0073] Specifically, a 4D CT image of the lungs can be selected as the experimental subject, with the end of expiration serving as the second floating image and all other phases serving as reference images. A 4D CT lung image includes 3D CT images from multiple respiratory phases. After acquiring projection data, 4D CT performs phase splitting based on the diaphragm position in the 2D projection data, typically into 8 to 10 phases. The projection data from each phase is then reconstructed separately to form a 3D CT image. The 3D CT images from different phases are then superimposed to form the 4D CT image.

[0074] Step 202: performing data expansion based on the second floating image to obtain 3D CT images of each expiratory phase;

[0075] During implementation, data expansion is performed based on the second floating image to obtain 3D CT images of each expiratory phase. This can be done by obtaining deformation fields between multiple phases based on the floating image; superimposing the deformation fields within a phase with the deformation fields between multiple phases to obtain multiple mixed deformation fields; and applying the multiple mixed deformation fields to the second floating image to obtain 3D CT images of each expiratory phase.

[0076] Among them, the deformation fields between multiple phases can be obtained by matching the other M phases through the second floating image, where M is 7 or 9; and two of the deformation fields between the M phases are randomly selected and superimposed with random weights to obtain multiple deformation fields between phases from the end of exhalation to the remaining phases.

[0077] For example, the respiratory end image in the 4D CT image is used as a phase, that is, as a floating image, and then a floating image is used to match the other 9 phases to obtain the deformation field between the phases; two of the obtained 9 phase deformation fields are randomly selected and randomly weighted superimposed to obtain a large number of phase deformation fields.

[0078] The deformation field within a phase can be obtained by setting random control points and random movement distances for the target phase. Based on the set random control points and random movement distances, thin plate spline interpolation simulation is used to obtain the deformation field within multiple phases of the target phase. In other words, a few control points are selected and the thin plate spline interpolation simulation algorithm is used to obtain the deformation field within a large number of phases.

[0079] Step 203: Projecting the obtained 3D CT images of each expiratory phase by light projection to obtain two 2D DRR (Digitally Reconstructed Radiographs) at orthogonal angles corresponding to each respiratory phase;

[0080] Step 204: A registration network model is obtained by performing model training on the two 2D DRRs at orthogonal angles corresponding to each respiratory phase and the second floating image.

[0081] When training the model, the features of the 2D DRR can be extracted using the residual block, and the channel layer is used as the third-dimensional feature to output a three-dimensional feature map; the three-dimensional feature map and the floating image are input into the 3D feature extraction network to obtain a three-dimensional deformation field; the three-dimensional deformation field is applied to the floating image through the spatial transformation layer to obtain a predicted 3D CT image corresponding to the 2D DRR; the maximum similarity between the predicted 3D CT image and the true 3D CT image obtained by data expansion is calculated to iteratively optimize the network model and train the registration network model.

[0082] Among them, in the process of model training, the mutual information between the prediction and the gold standard, the Dice similarity index between the segmentation labels, and the smoothness constraint of the deformation field can be used as the loss function. Specifically, the loss function can be expressed as:

[0083] L=λ1L Dice (F seg ,P seg )+λ2L MI (F CT ,P CT )+λ3L Reg (φ p )

[0084] Among them, F represents the true value, P represents the predicted value, n represents the number of categories in the image, i represents the i-th category of the image, p represents all elements in the entire deformation field, λ1, λ2, and λ3 represent L Dice , L MI , L Reg The weight, L Dice represents the loss between two segmentation labels, L MI Denotes the loss between two images, L Reg represents the regularization loss of the deformation field, F seg represents the segmentation label of the real 3D CT image, P seg Denotes the predicted segmentation label of the 3D CT image, F CT represents the real 3D CT image, P CT represents the predicted 3D CT image, φ p represents the predicted deformation field.

[0085] Specifically, during training, data augmentation can be performed first. An expiratory image from a specific phase of 4D CT is selected as the floating image. The remaining phases are then used as reference images for traditional image registration, yielding nine inter-phase deformation fields. Two of these are then randomly weighted and superimposed to generate a large number of deformation fields. Transient Prediction Scaling (TPS) is then used to simulate random intra-phase deformations. Inter-phase and intra-phase deformations are then superimposed with random weights to generate a large number of deformation fields representing each respiratory phase. These fields are then applied to the floating image to yield a large number of images (3D) representing each respiratory phase. Then, using a tracking method, light projections are applied to the enhanced 3D image to obtain the corresponding 2D dynamic range (DRR) at orthogonal angles. The resulting DRR and the floating image are then fed into the network for training. The network output is the 3D image at the corresponding moment of the predicted DRR, while the true value is known (obtained through the aforementioned data augmentation). The two are then maximized to complete model training by calculating their similarity.

[0086] When applying this model clinically, the floating image is the planning CT scan (pre-radiotherapy) used for radiotherapy planning, and the 2D DRR can be an X-ray or CBCT projection (during radiotherapy). During implementation, the planning CT scan and the 2D images during radiotherapy can be directly substituted into the trained model to generate 3D images during radiotherapy.

[0087] The above method is described below in conjunction with a specific embodiment. However, it should be noted that this specific embodiment is only for better illustrating the present application and does not constitute an improper limitation to the present application.

[0088] Considering that dimensionality can be reduced to 2D / 2D registration based on 2D / 3D registration, model training is completed by evaluating the similarity between DRR (Digitally Reconstructed Radiograph) and X-rays. However, this dimensionality reduction method will result in a certain degree of loss of spatial information. In addition, even with graphics processing unit (GPU) support, forward projection, back projection, and DRR generation are computationally expensive. In existing image registration, iterative optimization is required one by one during the testing phase, resulting in slow registration speed. Existing deep learning-based registration methods suffer from high dose (2D images are usually multiple and multi-angle) and long registration time.

[0089] To address the challenges of retrospective 3D / 3D registration, which struggles to meet the demands of real-time radiotherapy, and the high radiation dose and lengthy registration times associated with 2D / 3D registration, this study proposes a 2D / 3D non-rigid registration algorithm based on orthogonal angles. This algorithm allows for rapid and accurate 3D lung tumor tracking, using only two 2D X-ray projections from the radiotherapy session. Because the 2D images used are only two orthogonal 2D X-ray projections from the radiotherapy session, radiation dose can be effectively reduced, accelerating the radiotherapy process while maintaining high organ and tumor tracking accuracy.

[0090] Specifically, a method for 2D / 3D image registration for lung tumor tracking is provided, which may include the following steps:

[0091] S1: Hybrid Data Augmentation:

[0092] Considering that paired 2D / 3D medical images at the same moment are relatively scarce, a large number of 3D CT images representing various respiratory stages of the lungs can be obtained through data augmentation.

[0093] In implementation, traditional image registration Demons can be used to register sequentially to obtain inter-phase deformations φ1,...i,j. Then, by randomly extracting two random weights and superimposing them, a large number of inter-phase deformations from the end of expiration to the remaining phases can be obtained. Then, using thin plate spline interpolation simulation, by setting random control points and random movement distances, a large number of deformations simulating a certain phase can be obtained. Then, by superimposing the inter-phase deformations and the intra-phase deformations with random weights, a large number of mixed deformation fields can be obtained. Then, by applying them to the end of expiration, a 3D CT F field representing each expiratory phase of the lung can be achieved. CT .

[0094] That is, in this example, the 4D CT image of the lungs is selected as the experimental object, and the end of expiration is used as the floating image M CT The remaining phases serve as reference images. 4D CT lung images include 3D CT images from multiple respiratory phases. After acquiring projection data, 4D CT performs phase splitting based on the diaphragm position in the 2D projection data, typically into 8 to 10 phases. The projection data from each phase is then reconstructed separately to form a 3D CT image. The 3D CT images from different phases are superimposed to form the 4D CT image.

[0095] S2: Digital reconstruction image generation:

[0096] Specifically, the augmented data above represents a 3D CT image during radiotherapy. Model training requires the 2D DRR (Digitally Reconstructed Radiograph) at the corresponding moment. Therefore, in this example, a light projection method is used to project the 3D CT image, generating a 2D DRR at an orthogonal angle at the corresponding moment to simulate a real 2D X-ray projection for model training.

[0097] That is, through the light projection method, the 3D CT data representing each respiratory stage of the lungs after data expansion is projected to obtain two 2D DRRs at orthogonal angles at the corresponding moments to simulate the real X-ray projection in radiotherapy.

[0098] S3: 2D / 3D registration network:

[0099] For 2D / 3D registration, we first need to consider the consistency of spatial dimensions. In this case, the dimensionality increase method is used to transform the 2D / 3D registration problem into a 3D / 3D registration problem. First, the residual block is used to extract the features of the 2D DRR, and the channel layer is used as the third dimension feature to output a three-dimensional feature map. The three-dimensional feature map and the floating image (i.e., the end-expiratory MCT image with data expansion) are then input into the 3D feature extraction network. Among them, the 3D feature extraction network is based on the Attention Unet 3D / 3D registration network. Under the joint action of the encoding and decoding mechanism and the attention mechanism, the network outputs a three-dimensional deformation field φ p Then, the three-dimensional deformation field is applied to the floating image through the spatial transformation layer to obtain the corresponding 3D CT image PCT corresponding to the predicted DRR (PCT is the image output by the model and also the predicted 3DCT corresponding to the 2D DRR, FCT is the prediction). And the true value F CT (FCT is the true 3D CT obtained by expanding the mixed data in S1, which is the true value) is known. By calculating the maximum similarity between the predicted value and the true value and continuously iterating and optimizing, the model training can be completed.

[0100] In the inference stage, it is only necessary to input an end-expiratory image MCT based on orthogonal angles during radiotherapy and another 2D X-ray projection or DRR into the network to obtain a 3D image at the time of radiotherapy.

[0101] Among them, during the model training process, the mutual information between the prediction and the gold standard, the Dice similarity index between the segmentation labels, and the smoothness constraint of the deformation field are used as the loss function:

[0102] L=λ1L Dice (F seg,P seg )+λ2L MI (F CT ,P CT )+λ3L Reg (φ p )

[0103] Among them, F represents the true value, P represents the predicted value, n represents the number of categories in the image, i represents the i-th category of the image, p represents all elements in the entire deformation field, λ1, λ2, and λ3 represent L Dice , L MI , L Reg The weight, L Dice represents the loss between two segmentation labels, L MI Denotes the loss between two images, L Reg represents the regularization loss of the deformation field, F seg represents the segmentation label of the real 3D CT image, P seg Indicates the predicted segmentation label of the 3D CT image. In this example, λ1, λ2, and λ3 can be set to 0.5, 0.5, and 0.1, respectively. Seg represents the segmentation label, F represents the true value, which is the gold standard, P represents the predicted value, and φ p represents the predicted deformation field.

[0104] In the above example, a dimensionality-increasing approach was employed, using the channel layer as the third-dimensional feature, transforming the 2D / 3D registration problem into a 3D / 3D registration problem. Leveraging residual blocks and an attention mechanism, non-rigid 2D / 3D registration based on orthogonal angles was achieved. Through hybrid data augmentation, digitally reconstructed image generation, and 2D / 3D image registration, rapid 3D lung tumor tracking was achieved at a low radiation dose. This means that 3D tumor tracking can be achieved with high accuracy using only two orthogonal X-ray projections, significantly reducing the patient's radiation dose and simplifying clinical hardware.

[0105] In this application, 2D / 3D registration is used, but the registration needs to ensure that the two images are in the same spatial dimension. Therefore, the 2D / 3D can be reduced to 2D / 2D or upgraded to 3D / 3D. The existing method is mainly to reduce the dimension to 2D / 2D, but this will lead to the loss of spatial information. Therefore, in this example, the dimension is upgraded to 3D / 3D, which can alleviate the loss of spatial information. For 2D images, in this example, only two projections based on orthogonal angles are selected, which further reduces the irradiation dose. This method can be used to study the changes in elastic organs with respiratory movement and to realize tumor tracking based on orthogonal angle X-ray projections during radiotherapy.

[0106] During implementation, the ultimate goal of registering the real patient's tumor and the tumor represented by the phantom is to align the two images before and after registration, that is, to obtain a 3D image during radiotherapy by using the image used by the doctor to formulate the radiotherapy plan before radiotherapy.

[0107] Taking a specific example, as shown in Figure 3, the 2D / 3D non-rigid image registration method for lung tumor tracking includes the following process:

[0108] a) Training process:

[0109] First, a large number of CT FCT and segmentation Fseg representing each respiratory phase are obtained by hybrid data enhancement of the moving image MCT and the corresponding segmentation image Mseg.

[0110] Then, the FCT image is projected to obtain two-dimensional DRR90 and DRR00, which are input into the registration network together with the floating image MCT, which outputs the predicted deformation field φ p , and φ p Acting on the floating image MCT and the floating segmentation map Mseg, the corresponding predicted images PCT and Pseg are obtained.

[0111] b) Hybrid data augmentation.

[0112] c) Reasoning stage.

[0113] In reasoning, only two X-rays or CBCT projections at orthogonal angles during radiotherapy and the PCT image of the specified plan before radiotherapy need to be input to obtain the 3D organs and target areas during radiotherapy.

[0114] Based on the above method, experiments were conducted on a dynamic lung phantom from the end of exhalation to the end of inspiration using patient data with tumors. The experimental results show that both real patient and phantom experiments achieved good tumor tracking results. In other words, the method provided in this example can quickly complete 3D organ and tumor tracking using only two orthogonal X-ray angles.

[0115] The method embodiments provided in the above embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on an electronic device as an example, Figure 4 is a hardware structure block diagram of an electronic device for an image registration method for tumor tracking provided by the present application. As shown in Figure 4, the electronic device 10 may include one or more (only one is shown in the figure) processors 02 (the processor 02 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 04 for storing data, and a transmission module 06 for communication functions. It can be understood by those skilled in the art that the structure shown in Figure 4 is only for illustration and does not limit the structure of the above-mentioned electronic device. For example, the electronic device 10 may also include more or fewer components than those shown in Figure 4, or have a configuration different from that shown in Figure 4.

[0116] The memory 04 can be used to store software programs and modules of application software, such as the program instructions / modules corresponding to the image registration method for tumor tracking in the embodiment of the present application. The processor 02 executes various functional applications and data processing by running the software programs and modules stored in the memory 04, that is, implementing the image registration method for tumor tracking of the above-mentioned application. The memory 04 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 04 may further include a memory remotely located relative to the processor 02, and these remote memories may be connected to the electronic device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0117] The transmission module 06 is used to receive or send data via a network. Specific examples of the aforementioned network may include a wireless network provided by the communication provider of the electronic device 10. In one embodiment, the transmission module 06 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission module 06 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0118] At the software level, the image registration device for tumor tracking may be shown in FIG5 , and may include:

[0119] A first acquisition module 501 is used to acquire a respiratory end image;

[0120] The second acquisition module 502 is used to acquire two 2D images of the target phase based on orthogonal angles;

[0121] The registration module 503 is used to input the respiratory end image and the two 2D images based on orthogonal angles into the registration network model to obtain a 3D registered image for tumor tracking during radiotherapy.

[0122] In one embodiment, the registration network model can be established as follows:

[0123] acquiring a respiratory image at a predetermined phase in a 4D CT image as a floating image;

[0124] Performing data expansion based on the floating image to obtain 3D CT images of each expiratory phase;

[0125] By projecting light, the obtained 3D CT images of each expiratory phase are projected to obtain two 2D DRRs at orthogonal angles corresponding to each respiratory phase;

[0126] The registration network model is obtained by performing model training on the two 2D DRRs at orthogonal angles corresponding to each respiratory phase and the floating image.

[0127] In one embodiment, the registration network model is obtained by performing model training on the two 2D DRRs at orthogonal angles corresponding to each respiratory phase and the floating image, which may include:

[0128] The residual block is used to extract the features of 2D DRR, and the channel layer is used as the third dimension feature to output a three-dimensional feature map;

[0129] Inputting the three-dimensional feature map and the floating image into a 3D feature extraction network to obtain a three-dimensional deformation field;

[0130] Applying the three-dimensional deformation field to the floating image through a spatial transformation layer to obtain a predicted 3D CT image corresponding to the 2D DRR;

[0131] The maximum similarity between the predicted 3D CT image and the real 3D CT image obtained by data expansion is calculated to iteratively optimize the network model and train the registration network model.

[0132] In one embodiment, in calculating the maximum similarity between the predicted 3D CT image and the true 3D CT image obtained by data augmentation to iteratively optimize the network model and train the registration network model, the following formula can be used as the loss function:

[0133] L=λ1L Dice (F seg ,P seg )+λ2L MI (FCT ,P CT )+λ3L Reg (φ p )

[0134] Among them, F represents the true value, P represents the predicted value, n represents the number of categories in the image, i represents the i-th category of the image, p represents all elements in the entire deformation field, λ1, λ2, and λ3 represent L Dice , L MI , L Reg The weight, L Dice represents the loss between two segmentation labels, L MI Denotes the loss between two images, L Reg represents the regularization loss of the deformation field, F seg represents the segmentation label of the real 3D CT image, P seg Denotes the predicted segmentation label of the 3D CT image, F CT represents the real 3D CT image, P CT represents the predicted 3D CT image, φ p represents the predicted deformation field.

[0135] In one embodiment, data expansion is performed based on the floating image to obtain 3D CT images of each expiratory phase, including:

[0136] obtaining a plurality of phase-to-phase deformation fields according to the floating image;

[0137] Multiple mixed deformation fields are obtained by superimposing the deformation fields within a phase on the deformation fields between multiple phases;

[0138] Applying multiple mixed deformation fields to the end-of-exhalation image to obtain a 3D CT image of each exhalation phase;

[0139] In one embodiment, obtaining a deformation field between a plurality of phases according to the floating image may include:

[0140] Matching other M phases through the floating image to obtain the deformation field between the M phases, where M is 7 or 9;

[0141] Two of the M inter-phase deformation fields are randomly selected and superimposed with random weights to obtain multiple inter-phase deformations from the end of exhalation to the remaining phases.

[0142] In one embodiment, the intra-phase deformation can be obtained as follows: random control points and random movement distances of the target phase are set; based on the set random control points and random movement distances, thin plate spline interpolation simulation is used to obtain the deformation fields within multiple phases of the target phase.

[0143] In one embodiment, the two 2D images based on orthogonal angles may include: 2D X-ray projections / or 2D DRR.

[0144] In this example, a registration network model establishment device is also provided, which may include:

[0145] an acquisition module, configured to acquire a respiratory image of a predetermined phase in a 4D CT image as a floating image;

[0146] an expansion module, configured to perform data expansion based on the floating image to obtain 3D CT images of each expiratory phase;

[0147] The projection module is used to project the obtained 3D CT images of each expiratory phase by light projection to obtain two 2D DRRs at orthogonal angles corresponding to each respiratory phase;

[0148] The training module is used to obtain a registration network model by performing model training on the two 2D DRRs at orthogonal angles corresponding to each respiratory phase and the floating image.

[0149] In one embodiment, the training module can be specifically used to extract features of the 2D DRR using the residual block, and use the channel layer as the third-dimensional feature to output a three-dimensional feature map; the three-dimensional feature map and the floating image are jointly input into the 3D feature extraction network to obtain a three-dimensional deformation field; the three-dimensional deformation field is applied to the floating image through the spatial transformation layer to obtain a predicted 3D CT image corresponding to the 2D DRR; the maximum similarity between the predicted 3D CT image and the true 3D CT image obtained by data expansion is calculated to iteratively optimize the network model and train the alignment network model.

[0150] In one embodiment, data expansion is performed based on the floating image to obtain 3D CT images of each expiratory phase, which may include: obtaining deformation fields between multiple phases based on the floating image; obtaining multiple mixed deformation fields by superimposing the deformation fields within a phase with the deformation fields between multiple phases; and applying the multiple mixed deformation fields to the end-of-exhalation image to obtain 3D CT images of each expiratory phase.

[0151] The embodiments of the present application also provide a specific implementation of an electronic device capable of implementing all steps of the image registration method for tumor tracking in the above embodiments. The electronic device specifically includes the following: a processor, a memory, a communication interface, and a bus; wherein the processor, the memory, and the communication interface communicate with each other via the bus; the processor is configured to call a computer program in the memory, and when the processor executes the computer program, all steps of the image registration method for tumor tracking in the above embodiments are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0152] Step 1: Acquire the end-of-respiration image as the first floating image;

[0153] Step 2: Acquire two 2D images of the target phase based on orthogonal angles;

[0154] Step 3: Input the first floating image and the two 2D images based on orthogonal angles into the registration network model to obtain a 3D registered image for tumor tracking during radiotherapy.

[0155] As can be seen from the above description, the present embodiment acquires a respiratory endpoint image and two orthogonal 2D images, and then inputs these images into a registration network model to produce a 3D registered image for tumor tracking during radiotherapy. In the above example, image registration is achieved by selecting only two orthogonal projections, effectively reducing radiation dose and achieving high accuracy to meet real-time requirements.

[0156] Embodiments of the present application also provide a computer-readable storage medium capable of implementing all steps of the image registration method for tumor tracking in the above-mentioned embodiment. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the computer program implements all steps of the image registration method for tumor tracking in the above-mentioned embodiment. For example, when the processor executes the computer program, the following steps are implemented:

[0157] Step 1: Acquire the end-of-respiration image as the first floating image;

[0158] Step 2: Acquire two 2D images of the target phase based on orthogonal angles;

[0159] Step 3: Input the first floating image and the two 2D images based on orthogonal angles into the registration network model to obtain a 3D registered image for tumor tracking during radiotherapy.

[0160] As can be seen from the above description, the present embodiment acquires a respiratory endpoint image and two orthogonal 2D images, and then inputs these images into a registration network model to produce a 3D registered image for tumor tracking during radiotherapy. In the above example, image registration is achieved by selecting only two orthogonal projections, effectively reducing radiation dose and achieving high accuracy to meet real-time requirements.

[0161] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the hardware + program embodiments are generally similar to the method embodiments, so their description is relatively simple. For relevant portions, refer to the description of the method embodiments.

[0162] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0163] Although the present application provides method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-creative work. The order of steps listed in the embodiments is only one way of executing the steps among many steps and does not represent the only execution order. When the actual device or client product is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, in a parallel processor or multi-threaded processing environment).

[0164] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, an in-vehicle human-computer interaction device, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0165] Although the present specification embodiment provides the method operation steps as described in the embodiment or flow chart, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiment is only one way in the order of execution of many steps and does not represent a unique execution order. When the device or terminal product in practice is executed, it can be performed in sequence or in parallel according to the method shown in the embodiment or the accompanying drawings (such as a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "comprise", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements not only include those elements, but also include other elements not clearly listed, or also include elements inherent to such process, method, product or equipment. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or equipment including the elements.

[0166] For the convenience of description, the above devices are described in terms of functions divided into various modules. Of course, when implementing the embodiments of this specification, the functions of each module can be implemented in the same or multiple software and / or hardware, or the modules that implement the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0167] Those skilled in the art will also appreciate that, in addition to implementing the controller in pure computer-readable program code, it is entirely possible to implement the same functionality by logically programming the method steps in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, and the like. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered structures within the hardware component. Alternatively, the devices for implementing various functions can be considered both software modules implementing the method and structures within the hardware component.

[0168] The present application is described with reference to the flow chart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one flow chart flow or multiple flows and / or one box or multiple boxes of the block diagram.

[0169] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0170] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0171] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0172] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0173] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0174] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0175] Embodiments of this specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. Embodiments of this specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In distributed computing environments, program modules may be located in local and remote computer storage media, including storage devices.

[0176] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between the various embodiments can be referenced across them. Each embodiment focuses on the differences from the other embodiments. In particular, since the system embodiments are generally similar to the method embodiments, their description is relatively simple. For relevant parts, reference can be made to the description of the method embodiments. Throughout this specification, reference to the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the embodiments in this specification. In this specification, the schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and integrate the different embodiments or examples, and features of different embodiments or examples, described in this specification, without conflict.

[0177] The above description is merely an example of the embodiments of this specification and is not intended to limit the embodiments of this specification. For those skilled in the art, various modifications and variations of the embodiments of this specification are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of this specification shall be included within the scope of the claims of the embodiments of this specification.

Claims

1. An image registration method for tumor tracking, It is characterized in that The method comprises: acquiring an end-of-respiration image as a first floating image; Obtain two 2D images based on orthogonal angles of the target phase; The first floating image and the two 2D images based on orthogonal angles are input into a pre-trained registration network model to obtain a 3D registered image for tumor tracking during radiotherapy.

2. The method according to claim 1, It is characterized in that The registration network model is trained in the following manner: acquiring a respiratory image of a predetermined phase in the 4D CT image as a second floating image; Performing data expansion based on the second floating image to obtain a 3D CT image of each expiratory phase; By projecting light, the obtained 3D CT images of each expiratory phase are projected to obtain two 2D DRRs at orthogonal angles corresponding to each respiratory phase; The registration network model is obtained by performing model training on the two 2D DRRs at orthogonal angles corresponding to each respiratory phase and the second floating image.

3. The method according to claim 2, It is characterized in that The registration network model is obtained by performing model training on the two 2D DRRs at orthogonal angles corresponding to each respiratory phase and the second floating image, including: The residual block is used to extract the features of 2D DRR, and the channel layer is used as the third dimension feature to output a three-dimensional feature map; Inputting the three-dimensional feature map and the second floating image into a 3D feature extraction network to obtain a three-dimensional deformation field; Applying the three-dimensional deformation field to the second floating image through a spatial transformation layer to obtain a predicted 3D CT image corresponding to the 2D DRR; The maximum similarity between the predicted 3D CT image and the real 3D CT image obtained by data expansion is calculated to iteratively optimize the network model and train the registration network model.

4. The method according to claim 3, It is characterized in that In the process of calculating the maximum similarity between the predicted 3D CT image and the real 3D CT image obtained by data expansion, iteratively optimizing the network model and training the registration network model, the following formula is used as the loss function: L=λ 1 L Dice (F seg ,P seg )+λ 2 L MI (F CT ,P CT )+λ 3 L Reg (φ p ) Among them, F represents the true value, P represents the predicted value, n represents the number of categories in the image, and i represents the i-th image. categories, p represents all elements in the entire deformation field, λ 1 , 2 , 3 Respectively represent L Dice , L MI , L Reg The weight, L Dice represents the loss between two segmentation labels, L MI represents the loss between two images, L Reg represents the regularization loss of the deformation field, F seg represents the segmentation label of the real 3D CT image, P seg represents the predicted segmentation label of the 3D CT image, F CT represents the real 3D CT image, P CT represents the predicted 3D CT image, φ p represents the predicted deformation field.

5. The method according to claim 2, It is characterized in that Data expansion is performed based on the second floating image to obtain 3D CT images of each expiratory phase, including: Obtaining a deformation field between a plurality of phases according to the second floating image; Multiple mixed deformation fields are obtained by superimposing deformation fields within a phase on deformation fields between multiple phases; A plurality of mixed deformation fields are applied to the end-of-exhalation image to obtain a 3D CT image of each exhalation phase.

6. The method according to claim 5, It is characterized in that Obtaining a plurality of deformation fields between phases according to the second floating image, including: Matching other M phases through the second floating image to obtain a deformation field between the M phases, wherein M is 7 or 9; Two of the M inter-phase deformation fields are randomly selected for random weight superposition to obtain multiple inter-phase deformation fields from the end of exhalation to the remaining phases.

7. The method according to claim 5, It is characterized in that The phase internal deformation field is obtained as follows: Set the random control point and random movement distance of the target phase; According to the set random control points and random moving distances, thin plate spline interpolation simulation is used to obtain the deformation fields in multiple phases of the target phase.

8. The method according to claim 1, It is characterized in that The two 2D images based on orthogonal angles include: 2D X-ray projection / or 2D DRR.

9. A method for establishing a registration network model, It is characterized in that include: acquiring a respiratory image of a predetermined phase in the 4D CT image as a floating image; Performing data expansion based on the floating image to obtain a 3D CT image of each expiratory phase; By projecting light, the obtained 3D CT images of each expiratory phase are projected to obtain two 2D DRRs at orthogonal angles corresponding to each respiratory phase; The registration network model is obtained by performing model training on the two 2D DRRs at orthogonal angles corresponding to each respiratory phase and the floating image.

10. The method according to claim 9, It is characterized in that The registration network model is obtained by training the two 2D DRRs at orthogonal angles corresponding to each respiratory phase and the floating image, including: The residual block is used to extract the features of 2D DRR, and the channel layer is used as the third dimension feature to output a three-dimensional feature map; Inputting the three-dimensional feature map and the floating image into a 3D feature extraction network to obtain a three-dimensional deformation field; Applying the three-dimensional deformation field to the floating image through a spatial transformation layer to obtain a predicted 3D CT image corresponding to the 2D DRR; The maximum similarity between the predicted 3D CT image and the real 3D CT image obtained by data expansion is calculated to iteratively optimize the network model and train the registration network model.

11. The method according to claim 9, It is characterized in that Data expansion is performed based on the floating image to obtain 3D CT images of each expiratory phase, including: Obtaining deformation fields between multiple phases according to the floating image; Multiple mixed deformation fields are obtained by superimposing deformation fields within a phase on deformation fields between multiple phases; A plurality of mixed deformation fields are applied to the end-of-exhalation image to obtain a 3D CT image of each exhalation phase.

12. An image registration device for tumor tracking, It is characterized in that include: A first acquisition module, used for acquiring a respiratory end image as a first floating image; The second acquisition module is used to acquire two 2D images of the target phase based on orthogonal angles; The registration module is used to input the first floating image and the two 2D images based on orthogonal angles into a pre-trained registration network model to obtain a 3D registered image for tumor tracking during radiotherapy.

13. A terminal device comprising a processor and a memory for storing instructions executable by the processor, It is characterized in that When the processor executes the instructions, the steps of the method according to any one of claims 1 to 8 are implemented.

14. A computer-readable storage medium having a computer program / instructions stored thereon, It is characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images