Deep learning method for simulating lung respiratory movement

Abstract

The invention discloses a deep learning method for simulating lung respiratory movement, which comprises the following steps of: constructing a respiratory movement learning model based on a Seq2Seq encoder decoder, and adopting a 4D lung respiratory medical image comprising a plurality of phase stages in a training data set of the respiratory movement learning model; obtaining a projection image in the coronal plane slice direction of the 3D lung respiration scanning image, and extracting feature points of multiple feature items from the projection image by using a 3DCNN convolutional neural network; according to the extracted feature points of the multi-feature item and a convolutional layer in the respiratory motion learning model, extracting features of a corresponding phase stage from the 4D lung respiratory medical image data, and splicing the features to form an auxiliary input quantity; and coding and decoding the auxiliary input quantity through a ConvLSTM layer in the respiratory movement learning model to generate a simulated lung respiratory movement prediction image.

Description

Technical Field

[0001] The present invention relates to the technical field of medical diagnosis and image-guided treatment, and in particular to a deep learning method for simulating lung respiratory movement. Background Art

[0002] In modern medical diagnostic procedures, respiratory motion impedes accurate diagnosis as well as image-guided treatment. For example, during radiotherapy, it can lead to poor local tumor control and increased radiation toxicity to normal organs. It can also appear as motion artifacts in acquired images, making it difficult to distinguish nodule / tumor morphological changes from those caused by respiratory motion.

[0003] To simulate respiratory motion, most existing methods use intuitive motion representations, such as time-varying displacement vector fields (DVFs) obtained by deformable image registration (DIR) in 4D images acquired during the respiratory cycle. Alternative drive methods use DVF as a function of the alternative respiratory signal. A representative technical solution is to adjust this simulated motion by using any 1D respiratory trajectory as an auxiliary input to create large changes. The respiratory phases of multiple future time points and their 3D DVFs can be predicted simultaneously.

[0004] However, when the 1D breathing trajectory is used as an auxiliary input to adjust this simulated lung breathing movement, only the amplitude of the diaphragm top movement is considered as a substitute for the breathing signal. It contains fewer features and less information, which results in poor results and low accuracy in the final simulated lung breathing movement.

[0005] In summary, the existing 1D respiratory trajectory is used as an auxiliary input to adjust the predicted respiratory phase in this simulated lung respiratory movement mode and the structural similarity index SSIM index value and peak signal-to-noise ratio PSNR of the real 4D CT image are low, and the model prediction performance needs to be improved. Summary of the invention

[0006] The purpose of the present invention is to provide a deep learning method for simulating lung respiratory movement to solve the technical problem in the prior art that the structural similarity index SSIM index value and the peak signal-to-noise ratio PSNR of the predicted respiratory phase and the real 4D CT image in this simulated lung respiratory movement mode are low due to the existing 1D respiratory trajectory used as an auxiliary input to adjust the existing 1D respiratory trajectory.

[0007] In order to solve the above technical problems, the present invention specifically provides the following technical solutions:

[0008] A deep learning method for simulating lung breathing motion, comprising:

[0009] Constructing a respiratory motion learning model based on Seq2Seq encoder-decoder, wherein a training data set of the respiratory motion learning model adopts a 4D lung respiratory medical image containing multiple phase stages;

[0010] Acquire a projection image of a 3D lung respiratory scan in a coronal slice direction, and use a 3DCNN convolutional neural network to extract feature points of multiple feature items from the projection image;

[0011] Extracting features of corresponding phases from the 4D lung respiratory medical image data according to the extracted feature points of the multiple feature items and the convolutional layer in the respiratory movement learning model, and splicing them to form an auxiliary input;

[0012] The auxiliary input is encoded and decoded through the ConvLSTM layer in the respiratory movement learning model to generate a simulated lung respiratory movement prediction image.

[0013] As a preferred solution of the present invention, 4D lung respiratory medical image data containing multiple phase stages are read, and the dimension of each 4D lung respiratory medical image data is set, and the initial stage image in the 4D lung respiratory medical image data of multiple phase stages is extracted, and the initial segment image is input into the respiratory motion learning model together.

[0014] As a preferred solution of the present invention, in extracting feature points of multiple feature items of the projection image using a 3DCNN convolutional neural network, the multiple feature items include but are not limited to the top of the diaphragm, the edge of the tumor, and the position of the trachea.

[0015] As a preferred solution of the present invention, the auxiliary input is encoded and decoded using the ConvLSTM layer in the Seq2Seq encoder-decoder model to generate a predicted deformation vector field DVF;

[0016] The spatial transformation layer in the Seq2Seq encoder-decoder model and the predicted deformation vector field DVF are used to deform the initial phase image into the predicted phase images of different breathing phases.

[0017] As a preferred solution of the present invention, the features of the corresponding phase stage are extracted from the 4D lung respiratory medical image data according to the extracted feature points of the multiple feature items and the convolution layer in the respiratory movement learning model, and the specific calculation formula of the auxiliary input quantity is constructed as follows:

[0018] n(It,dt)=dt+It, where n(It,dt) is the auxiliary input, It is the hidden state encoded from the image sequence in the initial stage to its stage t, and dt is the feature of the feature point of the multi-feature item extracted by the 3D convolutional neural network.

[0019] As a preferred solution of the present invention, the ConvLSTM layer in the Seq2Seq encoder-decoder model includes two stacked ConvLSTM3D layers.

[0020] As a preferred solution of the present invention, the convolution layer uses a kernel size of 3×3×3 and a stride size of 2×2×2, and reduces the matrix dimension of the convolution layer to 1 / 8.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] The present invention is based on 2D images as auxiliary input. Compared with the 1D respiratory motion tracking input, it has more information parameters to characterize the respiratory motion. Therefore, the structural similarity index measurement SSIM index value and peak signal-to-noise ratio PSNR of the predicted respiratory phase and the real 4DCT image are higher, and the performance is better. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0024] Figure 1 A schematic flow chart of a deep learning method for simulating lung respiratory motion according to an embodiment of the present invention;

[0025] Figure 2 This is a schematic diagram of the workflow of a model using 2D images as auxiliary input based on a Seq2Seq encoder-decoder model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] like Figure 1 and Figure 2 As shown, the present invention provides a deep learning method for simulating lung breathing motion, comprising:

[0028] Constructing a respiratory motion learning model based on Seq2Seq encoder-decoder, wherein a training data set of the respiratory motion learning model adopts a 4D lung respiratory medical image containing multiple phase stages;

[0029] Acquire a projection image of a 3D lung respiratory scan in a coronal slice direction, and use a 3DCNN convolutional neural network to extract feature points of multiple feature items from the projection image;

[0030] Extracting features of corresponding phases from the 4D lung respiratory medical image data according to the extracted feature points of the multiple feature items and the convolutional layer in the respiratory movement learning model, and splicing them to form an auxiliary input;

[0031] The auxiliary input is encoded and decoded through the ConvLSTM layer in the respiratory movement learning model to generate a simulated lung respiratory movement prediction image.

[0032] The 4D lung respiratory medical image data including multiple phase stages are read, and the dimension of each 4D lung respiratory medical image data is set, and the initial stage image in the 4D lung respiratory medical image data of multiple phase stages is extracted, and the initial segment image is input into the respiratory motion learning model.

[0033] In extracting feature points of multiple feature items from the projection image using a 3DCNN convolutional neural network, the multiple feature items include but are not limited to the top of the diaphragm, the edge of the tumor, and the position of the trachea.

[0034] Encode and decode the auxiliary input using the ConvLSTM layer in the Seq2Seq encoder-decoder model to generate a predicted deformation vector field DVF;

[0035] The spatial transformation layer in the Seq2Seq encoder-decoder model and the predicted deformation vector field DVF are used to deform the initial phase image into the predicted phase images of different breathing phases.

[0036] According to the extracted feature points of the multiple feature items and the convolution layer in the respiratory movement learning model, the features of the corresponding phase stage are extracted from the 4D lung respiratory medical image data, and the specific calculation formula of the auxiliary input quantity is constructed as follows:

[0037] n(It,dt)=dt+It, where n(It,dt) is the auxiliary input, It is the hidden state encoded from the image sequence in the initial stage to its stage t, and dt is the feature of the feature point of the multi-feature item extracted by the 3D convolutional neural network.

[0038] In the specific implementation of the above-mentioned specific implementation mode, by:

[0039] Step 10, reading 4D lung respiratory medical image data containing multiple phase stages, and setting the dimension of each 4D lung respiratory medical image data, and extracting a set number of 2D coronal slices from the 4D lung respiratory medical image data, and extracting the initial stage image from the 4D lung respiratory medical image data of multiple phase stages, wherein the size dimension of each slice is specifically height×width×number of phase stages, preferably 128×128×10;

[0040] Step 20: Use the convolutional layer in the Seq2Seq encoder-decoder model to perform feature encoding on the 4D lung respiratory medical image data to generate a first feature map. The size of the first feature map is 64×64×64×96, that is, depth×width×height×channel.

[0041] A 3D convolutional neural network is used to extract features of each 2D coronal slice at K phase stages to generate a second feature map, wherein the size of the second feature map is 1×64×64×96, i.e., depth×width×height×channel;

[0042] Step 30: splicing the first feature map extracted by the convolution layer and the second feature map extracted by the 3D convolutional neural network in the feature channel dimension to generate a spliced ​​feature map, wherein the size of the spliced ​​feature map is 64×64×64×96;

[0043] Step 40: Encode and decode the fused feature map using the ConvLSTM layer in the Seq2Seq encoder-decoder model to generate a predicted deformation vector field DVF;

[0044] Step 50: Use the spatial transformation layer in the Seq2Seq encoder-decoder model and the predicted deformation vector field DVF to deform the initial stage image into predicted stage images of different breathing stages.

[0045] The following table lists the SSIM results of the predicted phase images of the respiratory phase obtained in different ways. The experimental methods used are 4D-CT images without any auxiliary input, 4D-CT images based on the amplitude of the 1D respiratory trajectory as auxiliary input, and 4D-CT images based on the image features of 2D Coronal Images as auxiliary input. It can be seen from the figure that the structural similarity index SSIM value of the method of the present invention is better, and the peak signal-to-noise ratio PSNR is higher. The structural similarity index SSIM of the simulation results without any auxiliary input or using 1D respiratory trajectory auxiliary input is poor, and the peak signal-to-noise ratio PSNR value is lower.

[0046] method SSIM PSNR 4D-CT 0.86±0.054 35.97±2.36 Amplitude of 1D respiratory trajectory + 4D-CT 0.88±0.050 36.55±2.46 2D image features + 4D-CT 0.91±0.041 41.54±2.87

[0047] The above embodiments are only exemplary embodiments of the present application and are not intended to limit the present application. The protection scope of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and protection scope of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present application.

Claims

1. A deep learning method for simulating lung breathing motion, characterized in that: include, Constructing a respiratory motion learning model based on Seq2Seq encoder-decoder, wherein a training data set of the respiratory motion learning model adopts a 4D lung respiratory medical image containing multiple phase stages; Acquire a projection image of a 3D lung respiratory scan in a coronal slice direction, and use a 3DCNN convolutional neural network to extract feature points of multiple feature items from the projection image; Extracting features of corresponding phases from the 4D lung respiratory medical image data according to the extracted feature points of the multiple feature items and the convolutional layer in the respiratory movement learning model, and splicing them to form an auxiliary input; The auxiliary input is encoded and decoded through the ConvLSTM layer in the respiratory movement learning model to generate a simulated lung respiratory movement prediction image.

2. A deep learning method for simulating lung breathing motion according to claim 1, characterized in that: The 4D lung respiratory medical image data including multiple phase stages are read, and the dimension of each 4D lung respiratory medical image data is set, and the initial stage image in the 4D lung respiratory medical image data of multiple phase stages is extracted, and the initial segment image is input into the respiratory motion learning model.

3. A deep learning method for simulating lung breathing motion according to claim 1, characterized in that: In extracting feature points of multiple feature items from the projection image using a 3DCNN convolutional neural network, the multiple feature items include but are not limited to the top of the diaphragm, the edge of the tumor, and the position of the trachea.

4. A deep learning method for simulating lung breathing motion according to claim 1, characterized in that: Encode and decode the auxiliary input using the ConvLSTM layer in the Seq2Seq encoder-decoder model to generate a predicted deformation vector field DVF; The spatial transformation layer in the Seq2Seq encoder-decoder model and the predicted deformation vector field DVF are used to deform the initial phase image into the predicted phase images of different breathing phases.

5. A deep learning method for simulating lung breathing motion according to claim 1, characterized in that: According to the extracted feature points of the multiple feature items and the convolution layer in the respiratory movement learning model, the features of the corresponding phase stage are extracted from the 4D lung respiratory medical image data, and the specific calculation formula of the auxiliary input quantity is constructed as follows: n(It,dt)=dt+It, where n(It,dt) is the auxiliary input, It is the hidden state encoded from the image sequence in the initial stage to its stage t, and dt is the feature of the feature point of the multi-feature item extracted by the 3D convolutional neural network.

6. A deep learning method for simulating lung breathing motion according to claim 1, characterized in that: The ConvLSTM layer in the Seq2Seq encoder-decoder model consists of two stacked ConvLSTM3D layers.

7. A deep learning method for simulating lung breathing motion according to claim 1, characterized in that: The convolutional layer uses a kernel size of 3×3×3 and a stride size of 2×2×2, and reduces the matrix dimension of the convolutional layer to 1 / 8.

Images