A deep learning method for simulating lung breathing motion

Abstract

The present invention discloses a deep learning method for simulating lung respiratory movement, comprising constructing a respiratory movement learning model based on a Seq2Seq encoder-decoder, wherein a training data set of the respiratory movement learning model adopts a 4D lung respiratory medical image containing multiple phase stages; obtaining a projection image in the coronal slice direction of a 3D lung respiratory scan image, and extracting feature points of multiple feature items from the projection image using a 3DCNN convolutional neural network; extracting features of corresponding phase stages from the 4D lung respiratory medical image data according to the extracted feature points of the multiple feature items and a convolution layer in the respiratory movement learning model, thereby forming an auxiliary input; and encoding and decoding the auxiliary input using a ConvLSTM layer in the respiratory movement learning model to generate a predicted image of simulated lung respiratory movement.

Description

Technical Field

[0001] The present invention relates to the technical field of medical diagnosis and image-guided therapy, and in particular to a deep learning method for simulating lung respiratory motion. Background Art

[0002] In modern medical diagnostics, respiratory motion hinders accurate diagnosis and image-guided treatment. For example, during radiotherapy, it can lead to poor local tumor control and increased radiation toxicity to normal organs. It can also manifest as motion artifacts in acquired images, making it difficult to distinguish changes in nodule / tumor morphology from those caused by respiratory motion.

[0003] To simulate respiratory motion, most existing approaches use intuitive motion representations, such as time-varying displacement vector fields (DVFs) obtained by deformable image registration (DIR) of 4D images acquired during the respiratory cycle. Surrogate-driven approaches use the DVF as a function of the surrogate respiratory signal. A representative approach uses arbitrary 1D respiratory trajectories as auxiliary input to modulate this simulated motion, creating large variations. It is possible to simultaneously predict respiratory phases and their 3D DVFs at multiple future time points.

[0004] However, when using 1D respiratory trajectory as an auxiliary input to adjust this simulated lung breathing movement mode, only the amplitude of the diaphragm top movement is considered as a substitute for the respiratory signal. It contains fewer features and less information, which results in poor results and low accuracy in the final simulated lung breathing movement.

[0005] In summary, the existing 1D respiratory trajectory is used as an auxiliary input to adjust the predicted respiratory phase in this simulated lung respiratory movement mode. The structural similarity index SSIM index value and peak signal-to-noise ratio PSNR of the real 4D CT image are relatively low, and the model prediction performance needs to be improved. Summary of the Invention

[0006] The purpose of the present invention is to provide a deep learning method for simulating lung respiratory motion to solve the technical problem in the prior art that the structural similarity index (SSIM) value and peak signal-to-noise ratio (PSNR) of the predicted respiratory phase and the real 4D CT image in this simulated lung respiratory motion mode are low due to the existing 1D respiratory trajectory being used as an auxiliary input to adjust the existing 1D respiratory trajectory.

[0007] In order to solve the above technical problems, the present invention specifically provides the following technical solutions:

[0008] A deep learning method for simulating lung breathing motion, including,

[0009] Constructing a respiratory motion learning model based on Seq2Seq encoder-decoder, wherein the training dataset of the respiratory motion learning model adopts 4D lung respiratory medical images containing multiple phase stages;

[0010] Obtaining a projection image of a coronal slice direction of a 3D lung respiratory scan image, and extracting feature points of multiple feature items from the projection image using a 3DCNN convolutional neural network;

[0011] Extracting features of corresponding phases from the 4D lung respiratory medical image data according to the extracted feature points of the multiple feature items and the convolution layer in the respiratory movement learning model, and splicing them to form an auxiliary input;

[0012] The auxiliary input is encoded and decoded through the ConvLSTM layer in the respiratory motion learning model to generate a simulated lung respiratory motion prediction image.

[0013] As a preferred solution of the present invention, 4D lung respiratory medical image data containing multiple phase stages is read, and the dimension of each 4D lung respiratory medical image data is set, and the initial stage image in the 4D lung respiratory medical image data of multiple phase stages is extracted, and the initial segment image is input into the respiratory motion learning model.

[0014] As a preferred solution of the present invention, in extracting feature points of multiple feature items from the projection image using a 3DCNN convolutional neural network, the multiple feature items include but are not limited to the top of the diaphragm, the edge of the tumor, and the position of the trachea.

[0015] As a preferred solution of the present invention, the auxiliary input is encoded and decoded using the ConvLSTM layer in the Seq2Seq encoder-decoder model to generate a predicted deformation vector field DVF;

[0016] The spatial transformation layer in the Seq2Seq encoder-decoder model and the predicted deformation vector field DVF are used to deform the initial phase image into the predicted phase images of different breathing phases.

[0017] As a preferred solution of the present invention, the features of the corresponding phase stage are extracted from the 4D lung respiratory medical image data according to the extracted feature points of the multiple feature items and the convolution layer in the respiratory movement learning model, and the specific calculation formula of the auxiliary input quantity is constructed as follows:

[0018] n(It,dt)=dt+It, where n(It,dt) is the auxiliary input, It is the hidden state encoded from the initial stage image sequence to its stage t, and dt is the feature of the feature point of the multi-feature item extracted by the 3D convolutional neural network.

[0019] As a preferred solution of the present invention, the ConvLSTM layer in the Seq2Seq encoder-decoder model includes two stacked ConvLSTM3D layers.

[0020] As a preferred solution of the present invention, the convolution layer uses a kernel size of 3×3×3 and a stride size of 2×2×2, and reduces the matrix dimension of the convolution layer to 1 / 8.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] The present invention is based on 2D images as auxiliary input. Compared with the 1D respiratory motion tracking input, it has more information parameters to characterize the respiratory motion. Therefore, the structural similarity index measurement SSIM index value and peak signal-to-noise ratio PSNR of the predicted respiratory phase and the real 4DCT image are higher, and the performance is better. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other implementation drawings based on the provided drawings without inventive effort.

[0024] Figure 1 Schematic diagram of a flow chart of a deep learning method for simulating lung respiratory motion according to an embodiment of the present invention;

[0025] Figure 2 Schematic diagram of the workflow of a 2D image as auxiliary input model based on a Seq2Seq encoder-decoder model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0027] like Figure 1 and Figure 2 As shown, the present invention provides a deep learning method for simulating lung respiratory motion, comprising:

[0028] Constructing a respiratory motion learning model based on Seq2Seq encoder-decoder, wherein the training dataset of the respiratory motion learning model adopts 4D lung respiratory medical images containing multiple phase stages;

[0029] Obtaining a projection image of a coronal slice direction of a 3D lung respiratory scan image, and extracting feature points of multiple feature items from the projection image using a 3DCNN convolutional neural network;

[0030] Extracting features of corresponding phases from the 4D lung respiratory medical image data according to the extracted feature points of the multiple feature items and the convolution layer in the respiratory movement learning model, and splicing them to form an auxiliary input;

[0031] The auxiliary input is encoded and decoded through the ConvLSTM layer in the respiratory motion learning model to generate a simulated lung respiratory motion prediction image.

[0032] 4D lung respiratory medical image data including multiple phase stages is read, and the dimension of each 4D lung respiratory medical image data is set, and the initial stage image in the 4D lung respiratory medical image data of multiple phase stages is extracted, and the initial segment image is input into the respiratory motion learning model.

[0033] In extracting feature points of multiple feature items from the projection image using a 3DCNN convolutional neural network, the multiple feature items include but are not limited to the top of the diaphragm, the edge of the tumor, and the position of the trachea.

[0034] Encode and decode the auxiliary input using the ConvLSTM layer in the Seq2Seq encoder-decoder model to generate a predicted deformation vector field DVF;

[0035] The spatial transformation layer in the Seq2Seq encoder-decoder model and the predicted deformation vector field DVF are used to deform the initial phase image into the predicted phase images of different breathing phases.

[0036] The features of the corresponding phase stage are extracted from the 4D lung respiratory medical image data according to the extracted feature points of the multiple feature items and the convolution layer in the respiratory movement learning model. The specific calculation formula of the auxiliary input quantity is constructed as follows:

[0037] n(It,dt)=dt+It, where n(It,dt) is the auxiliary input, It is the hidden state encoded from the initial stage image sequence to its stage t, and dt is the feature of the feature point of the multi-feature item extracted by the 3D convolutional neural network.

[0038] In the specific implementation of the above embodiment, by:

[0039] Step 10: Reading 4D lung respiratory medical image data comprising multiple phase stages, setting the dimensions of each 4D lung respiratory medical image data, extracting a set number of 2D coronal slices from the 4D lung respiratory medical image data, and extracting an initial phase image from the 4D lung respiratory medical image data of the multiple phase stages, wherein the dimensions of each slice are specifically height × width × number of phase stages, preferably 128 × 128 × 10;

[0040] Step 20: Use the convolutional layer in the Seq2Seq encoder-decoder model to perform feature encoding on the 4D lung respiratory medical image data to generate a first feature map. The size of the first feature map is 64×64×64×96, that is, depth×width×height×channels.

[0041] A 3D convolutional neural network is used to extract features of each 2D coronal slice at K phase stages to generate a second feature map. The size of the second feature map is 1×64×64×96, i.e., depth×width×height×channel.

[0042] Step 30: Splice the first feature map extracted by the convolutional layer and the second feature map extracted by the 3D convolutional neural network in the feature channel dimension to generate a spliced ​​feature map. The size of the spliced ​​feature map is 64×64×64×96.

[0043] Step 40: Use the ConvLSTM layer in the Seq2Seq encoder-decoder model to encode and decode the fused feature map to generate a predicted deformation vector field DVF;

[0044] Step 50: Use the spatial transformation layer in the Seq2Seq encoder-decoder model and the predicted deformation vector field DVF to deform the initial phase image into the predicted phase images of different breathing phases.

[0045] The table below lists the SSIM results of predicted respiratory phase images obtained using different methods, including 4D-CT images without any auxiliary input, 4D-CT images using the amplitude of a 1D respiratory trajectory as auxiliary input, and 4D-CT images using the image features of 2D Coronal Images as auxiliary input. As can be seen from the figure, the method of the present invention achieves a good SSIM value and a high PSNR. Simulation results without any auxiliary input or using a 1D respiratory trajectory as auxiliary input have poor SSIM values ​​and low PSNR values.

[0046] method SSIM PSNR 4D-CT 0.86±0.054 35.97±2.36 Amplitude of 1D respiratory trajectory + 4D-CT 0.88±0.050 36.55±2.46 2D image features + 4D-CT 0.91±0.041 41.54±2.87

[0047] The above embodiments are merely exemplary embodiments of the present application and are not intended to limit the scope of the present application. The scope of protection of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and scope of protection of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the scope of protection of the present application.

Claims

1. A deep learning method for simulating lung breathing motion, characterized in that: include, Constructing a respiratory motion learning model based on Seq2Seq encoder-decoder, wherein the training dataset of the respiratory motion learning model adopts 4D lung respiratory medical images containing multiple phase stages; Obtaining a projection image of a coronal slice direction of a 3D lung respiratory scan image, and extracting feature points of multiple feature items from the projection image using a 3DCNN convolutional neural network; Extracting features of corresponding phases from the 4D lung respiratory medical image data according to the extracted feature points of the multiple feature items and the convolution layer in the respiratory movement learning model, and splicing them to form an auxiliary input; encoding and decoding the auxiliary input through the ConvLSTM layer in the respiratory motion learning model to generate a simulated lung respiratory motion prediction image; Reading 4D lung respiratory medical image data comprising multiple phase stages, setting the dimension of each 4D lung respiratory medical image data, extracting an initial stage image from the 4D lung respiratory medical image data of the multiple phase stages, and inputting the initial segment image into the respiratory motion learning model; In extracting feature points of multiple feature items from the projection image using a 3DCNN convolutional neural network, the multiple feature items include but are not limited to the top of the diaphragm, the edge of the tumor, and the position of the trachea; The features of the corresponding phase stage are extracted from the 4D lung respiratory medical image data according to the extracted feature points of the multiple feature items and the convolution layer in the respiratory movement learning model. The specific calculation formula of the auxiliary input quantity is constructed as follows: n(It,dt)=dt+It, where n(It,dt) is the auxiliary input, It is the hidden state encoded from the initial stage image sequence to its stage t, and dt is the feature of the feature point of the multi-feature item extracted by the 3D convolutional neural network.

2. A deep learning method for simulating lung respiratory motion according to claim 1, characterized in that: Encode and decode the auxiliary input using the ConvLSTM layer in the Seq2Seq encoder-decoder model to generate a predicted deformation vector field DVF; The spatial transformation layer in the Seq2Seq encoder-decoder model and the predicted deformation vector field DVF are used to deform the initial phase image into the predicted phase images of different breathing phases.

3. A deep learning method for simulating lung respiratory motion according to claim 1, characterized in that: The ConvLSTM layer in the Seq2Seq encoder-decoder model consists of two stacked ConvLSTM3D layers.

4. A deep learning method for simulating lung respiratory motion according to claim 1, characterized in that: The convolutional layer uses a kernel size of 3×3×3 and a stride size of 2×2×2, and reduces the matrix dimension of the convolutional layer to 1 / 8.

Images