A CT ventilation function imaging method based on unsupervised learning

Abstract

The present invention provides a CT ventilation function imaging method based on unsupervised learning. Based on a convolutional neural network, an unsupervised registration module and a ventilation function imaging module are constructed, and the two constitute a CT ventilation function imaging network; an unsupervised loss function is constructed; a training set is prepared, and lung CT images of multiple subjects in two different respiratory states are used as input, and the CT ventilation function imaging network is unsupervisedly trained without any labels. CT ventilation function imaging is performed on a new subject, and the lung CT images of the new subject in two different respiratory states are input into the trained CT ventilation function imaging network, and the ventilation function image of the subject can be automatically output within a few seconds. The present invention can quickly realize ventilation function imaging of the subject, and is used for real-time evaluation of the local and overall ventilation function of the subject.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a CT ventilation function imaging method based on unsupervised learning, which is suitable for real-time evaluation of the local and overall ventilation function of a subject. Background Art

[0002] The lungs are important organs in the human respiratory system. Their main function is to take in oxygen from the air and expel carbon dioxide from the body. The gas-gas exchange process between the lungs and the external environment is called pulmonary ventilation. The ventilation function of the lungs measures the body's ability to inhale external air into the lungs and expel waste gas from the lungs. Ventilatory function imaging is a way to evaluate the ventilation function of a subject. Ventilatory function imaging refers to directly imaging the ventilation distribution in the human lungs. Usually, the subject is asked to inhale some special gas contrast agents, and then the distribution of these gases in the lungs is imaged using physical measurement methods to obtain the ventilation function image of the subject, thereby evaluating the subject's local and overall ventilation function [Li H, Zhao X, Wang Y, et al. Science Advances, 2021, 7(1): eabc8180].

[0003] CT ventilation function imaging is a ventilation function imaging method proposed in recent years [Hegi-Johnson F, Keall P, Barber J, et al. Medical Physics, 2017, 44 (8): 4045-4055]. It is based on image post-processing methods and can use the subject's lung CT image to generate ventilation function images. The method flow is as follows: first, collect the subject's lung CT images in two different respiratory states (named CT1 image and CT2 image respectively); then, use the lung image registration algorithm (such as the commonly used affine transformation, B-spline deformation algorithm, etc.) to perform spatial registration on the CT1 image and the CT2 image; finally, according to the ventilation imaging model, calculate the relative change of the density (i.e., CT value) of each lung voxel in the two respiratory states, which is used to reflect the amount of air inhaled or exhaled by the lung voxel, thereby characterizing the ventilation function. In CT ventilation function imaging, the air naturally breathed by the human body can be regarded as a special gas contrast agent, so this method does not require the use of additional gas contrast agents and has broad application prospects.

[0004] However, the existing conventional CT ventilation function imaging methods all rely on lung image registration algorithms [Kipritidis J, Tahir BA, Cazoulat G, et al. Medical Physics, 2019, 46 (3): 1198-1217]. However, lung image registration algorithms usually require a long calculation time. At conventional image resolution (e.g., 512×512×192), the calculation time can reach several hours [Wu Z, Rietzel E, Boldea V, et al. Medical Physics, 2008, 35 (2): 775-781]. Conventional CT ventilation function imaging methods cannot evaluate the ventilation function of subjects in real time. In addition, the accuracy of the lung image registration algorithm will also affect the accuracy of CT ventilation function imaging, and the accuracy of the lung image registration algorithm is easily affected by various factors (such as similarity measurement, spatial transformation model, etc.). Therefore, it is necessary to develop a new CT ventilation function imaging method that is independent of the lung image registration algorithm and can achieve real-time imaging of the subjects' ventilation function. Summary of the invention

[0005] In view of the deficiency that conventional CT ventilation function imaging methods must rely on lung image registration algorithms and cannot achieve real-time imaging, the present invention provides a CT ventilation function imaging method based on unsupervised learning, which does not require the use of lung image registration algorithms and can achieve real-time imaging of the ventilation function of a subject.

[0006] The above-mentioned object of the present invention is achieved by the following technical solutions:

[0007] The CT ventilation function imaging method based on unsupervised learning includes the following steps:

[0008] Step 1, collecting and preprocessing lung CT images of subjects in two different respiratory states, the lung CT images include CT1 images and CT2 images, and merging the preprocessed CT1 image and CT2 image of each subject into a dual-channel image;

[0009] Step 2, constructing a CT ventilation function imaging network, the CT ventilation function imaging network includes an unsupervised registration module and a ventilation function imaging module, the unsupervised registration module generates an initial spatial transformation and an initial deformation image according to the dual-channel image, and the ventilation function imaging module generates a final total spatial transformation, a final deformation image and a ventilation function image according to the initial spatial transformation;

[0010] Step 3: Construct the loss function L of the CT ventilation function imaging network.

[0011] L=L sim +λ*L T

[0012] Among them, L sim is the similarity measure between the CT2 image and the final deformed image, λ represents the weight of the constraint term of the spatial transformation, and L T is the constraint term of the final total space transformation;

[0013] Step 4: Prepare a training set for the CT ventilation function imaging network, where dual-channel images of different subjects constitute samples in the training set;

[0014] Step 5: Perform unsupervised training on the CT ventilation function imaging network through the training set to obtain the final parameters of the CT ventilation function imaging network.

[0015] Step 6: Acquire CT1 images and CT2 images of the new subject's lungs in two respiratory states, generate dual-channel images and input them into the CT ventilation function imaging network to obtain the ventilation function image of the new subject.

[0016] The pre-processing of step 1 as described above comprises the following steps:

[0017] Step 1.1, the resolution of all lung CT images is adjusted to the same resolution using an interpolation algorithm;

[0018] Step 1.2, adjusting the CT value of the lung CT image to display only the lung window;

[0019] Step 1.3: Merge the preprocessed CT1 image and CT2 image of each subject into a dual-channel image.

[0020] As described above, the unsupervised registration module of step 2 includes a downsampling stage, an upsampling stage, and a spatial deformation stage in sequence. The spatial deformation stage includes two convolutional layers. The first convolutional layer in the spatial deformation stage outputs an initial spatial transformation, and the second convolutional layer outputs an initial deformed image through a spatial deformation function according to the initial spatial transformation output by the first convolutional layer.

[0021] As described above, the unsupervised registration modules in step 2 are multiple and connected in series, and the input of the latter unsupervised registration module is the initial deformed image output by the previous unsupervised registration module.

[0022] As mentioned above, the convolution kernels of the same unsupervised registration module are the same, and the convolution kernels of different unsupervised registration modules are different.

[0023] As mentioned above, the ventilation function imaging module includes three convolutional layers, wherein the first convolutional layer merges the initial spatial transformations obtained by each unsupervised registration module to generate the final total spatial transformation; the second convolutional layer applies the final total spatial transformation to the CT1 image to generate the final deformed image; the third convolutional layer calculates the relative change of the CT value of each lung voxel in the CT2 image based on the CT2 image and the final deformed image to obtain the ventilation function image.

[0024] As mentioned above, the similarity measure is the mean square error between the CT2 image and the final deformed image, or the normalized cross-correlation coefficient between the CT2 image and the final deformed image, or the similarity measure is the mutual information between the CT2 image and the final deformed image.

[0025] As mentioned above, the constraint term L of the final total space transformation T Based on the following formula:

[0026]

[0027] Where p represents pixel, Ω represents image space, represents the gradient operator and T is the final total spatial transformation.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. By using the CT ventilation function imaging method based on unsupervised learning provided by the present invention, after acquiring the lung CT image of the subject, the ventilation function image of the subject can be calculated within a few seconds, thereby realizing real-time imaging of the ventilation function of the subject.

[0030] 2. By using the CT ventilation function imaging method based on unsupervised learning provided by the present invention, it is not necessary to manually set various parameters required for lung image registration and ventilation imaging model in conventional CT ventilation function imaging methods. Only the subject's lung CT1 image and CT2 image need to be input into the network, and the network can automatically output the ventilation function image, thus realizing automated imaging.

[0031] 3. The present invention constructs a CT ventilation function imaging network based on unsupervised learning, which does not require a lot of time and effort to make labels, saves time and effort, and can be quickly popularized and applied. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Flow chart of the method of the present invention.

[0033] Figure 2 This is the structural diagram of the CT ventilation functional imaging network.

[0034] Figure 3This is a comparison chart of the results of CT ventilation function imaging method based on unsupervised learning and conventional methods.

[0035] Figure 4 This is a time comparison chart between the CT ventilation function imaging method based on unsupervised learning and the conventional method. DETAILED DESCRIPTION

[0036] In order to facilitate those skilled in the art to understand and implement the present invention, the present invention is further described in detail below in conjunction with embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention and are not used to limit the present invention.

[0037] like Figure 1 As shown, a CT ventilation function imaging method based on unsupervised learning specifically includes the following steps:

[0038] Step 1: Design the input and output of the CT ventilation function imaging network. The input of the CT ventilation function imaging network is the lung CT images (including CT1 image and CT2 image) of the subject in two different respiratory states. Before entering the network, the lung CT images are preprocessed. First, the resolution of all lung CT images is adjusted to the same resolution using an interpolation algorithm. After weighing the hardware limitations of the computer and retaining as many image details as possible, the present invention sets the resolution of all lung CT images to 256×256×192. It should be noted that the specific value of the resolution can be adjusted according to actual conditions. Secondly, the CT value of the pixel in all lung CT images less than -1200HU is set to -1200HU, and the CT value of the pixel greater than 0HU is set to 0HU, that is, the CT value of the lung CT image is adjusted to an image that only displays the lung window, so that the CT ventilation function imaging network focuses on the lung area. Finally, the preprocessed CT1 image and CT2 image of each subject are merged into a dual-channel image, and the dual-channel image is input into the CT ventilation function imaging network. The output of the CT ventilation function imaging network is the ventilation function image. The CT ventilation function imaging network is unsupervised, so no labels are required.

[0039] Step 2: Build the structure of the CT ventilation function imaging network based on 3D convolutional neural network. The CT ventilation function imaging network includes an unsupervised registration module and a ventilation function imaging module. Its structure is as follows: Figure 2 As shown, the specific steps include:

[0040] Step 2.1: Construct an unsupervised registration module. The unsupervised registration module includes a downsampling stage, an upsampling stage, and a spatial deformation stage. In this embodiment, the downsampling stage of the unsupervised registration module has 4 convolutional layers, the upsampling stage has 7 convolutional layers, and the spatial deformation stage has 2 convolutional layers, such as Figure 2As shown. The convolution kernels of each convolution layer are of the same size, and zero-filling operation is used for convolution. Leaky ReLU activation function is used after each convolution layer. In the downsampling stage, the step size of the convolution kernel is set to 2, but in the upsampling stage and the spatial deformation stage, the step size of the convolution kernel is set to 1. The first convolution layer in the spatial deformation stage outputs the initial spatial transformation, and the second convolution layer uses the spatial deformation function to achieve image deformation according to the initial spatial transformation output by the first convolution layer, and outputs the initial deformed image. It should be noted that the specific number of layers and convolution kernel parameters of each stage can be adjusted according to actual conditions.

[0041] Step 2.2: Configure the number of unsupervised registration modules. Repeat the configuration of the unsupervised registration modules constructed in step 2.1 n times (the present invention takes n=2 as an example). In the first unsupervised registration module, a larger convolution kernel is used so that the network can capture larger deformations of the lungs; in the second unsupervised registration module, a smaller convolution kernel is used so that the network pays more attention to smaller local deformations of the lungs. Specifically, in the first unsupervised registration module, the size of the convolution kernel is set to 7×7×7, and in the second unsupervised registration module, the size of the convolution kernel is set to 3×3×3. The first unsupervised registration module and the second unsupervised registration module are connected in series, that is, the input of the second unsupervised registration module is the initial deformed image output by the first unsupervised registration module. Figure 2 In , T1 and T2 represent the initial spatial transformations corresponding to the first unsupervised registration module and the second unsupervised registration module, respectively, and W1 and W2 represent the initial deformed images corresponding to the first unsupervised registration module and the second unsupervised registration module, respectively. It should be noted that the specific value of n and the size of the convolution kernel can be adjusted according to actual conditions.

[0042] Step 2.3: Construct a ventilation function imaging module. The ventilation function imaging module contains three convolutional layers, such as Figure 2As shown. Among them, the first convolution layer generates the final total spatial transformation, and merges the n initial spatial transformations obtained by the n unsupervised registration modules to generate the final total spatial transformation. The second convolution layer generates the final deformable image based on the final total spatial transformation. The second convolution layer applies the final total spatial transformation to the CT1 image to generate the final deformable image. The third convolution layer generates the CT ventilation function image based on the CT2 image and the final deformable image. The third convolution layer calculates the relative change of the CT value of each lung voxel in the CT2 image based on the CT2 image and the final deformable image, which is used to characterize the ventilation function and obtain the ventilation function image. The size of the convolution kernel used in the ventilation function imaging module is 3×3×3, the step size of the convolution kernel is set to 1, the convolution adopts the zero-filling operation, and the Leaky ReLU activation function is used after each convolution layer. It should be noted that the parameters of the convolution kernel can be adjusted according to the actual situation.

[0043] Step 3: Construct an unsupervised loss function for the CT ventilation function imaging network. The loss function (defined as L) of the CT ventilation function imaging network (defined as f) consists of two parts: 1) the similarity measure (defined as L) between the CT2 image and the final deformed image (defined as W) sim ). In this embodiment, the mean square error is selected as the similarity measure; 2) The constraint term (defined as L) of the final total space transformation (defined as T) T ) is used to ensure that the spatial transformation has physical and physiological significance. This embodiment uses a gradient operator to constrain the final total spatial transformation. It should be noted that the similarity measure in the loss function and the constraint terms of the final total spatial transformation can be adjusted according to actual conditions, such as replacing the mean square error representing the similarity measure with a normalized mutual correlation coefficient or mutual information, and replacing the gradient operator representing the constraint term with a Jacobian determinant. Based on the above definition, the mathematical relationship between the final total spatial transformation T and the final deformable image W and the CT ventilation function imaging network f is:

[0044] T=f(CT1,CT2,θ) (1)

[0045]

[0046] Where T is the final total spatial transformation, f is the CT ventilation function imaging network, CT1 is the CT1 image, CT2 is the CT2 image, θ is the parameter of the CT ventilation function imaging network, W is the final deformed image, and the symbol ° indicates that the pixels in the CT1 image are displaced and interpolated according to the final total spatial transformation T to generate the final deformed image W. Based on formulas 1 and 2, given the CT1 image and the CT2 image, and the network parameter θ of the CT ventilation function imaging network, the final total spatial transformation T and the final deformed image W can be obtained. According to the above definition, the mathematical expression of the loss function L of the CT ventilation function imaging network is:

[0047] L=L sim +λ*L T (3)

[0048]

[0049]

[0050] Wherein, λ represents the weight of the constraint term of the spatial transformation (in this embodiment, λ=0.05 is set, but the specific value can be adjusted according to the actual situation), m represents the number of pixels in the CT2 image (the number of pixels in the CT2 image and the CT1 image is the same), p represents a pixel, Ω represents the image space, Represents the gradient operator.

[0051] Step 4: Create a training set for the CT ventilation function imaging network. Collect CT1 images and CT2 images of the subject's lungs in two respiratory states to generate dual-channel images for training. The dual-channel images of different subjects constitute the samples in the training set, and the training set can be created without corresponding labels.

[0052] Step 5: Perform unsupervised training on the CT ventilation function imaging network using the training set to obtain the final parameters of the CT ventilation function imaging network. The initial parameters of the CT ventilation function imaging network are randomly generated using the standard normal distribution function. The basic parameters for training the CT ventilation function imaging network are: batch size is 4; learning rate is 1×10 -5 The optimization algorithm uses the Adam optimizer; the number of training Epochs is 50. It should be noted that the basic parameters of the training can be adjusted according to the actual situation.

[0053] Step 6: Perform CT ventilation function imaging on the new subject. Collect CT1 and CT2 images of the new subject's lungs in two respiratory states, perform the preprocessing described in step 1 (this part takes a very short time), and then input them into the trained CT ventilation function imaging network. The CT ventilation function imaging network can automatically output the ventilation function image of this subject within a few seconds, which is used to evaluate the local and overall ventilation function of the subject in real time.

[0054] Figure 3 The ventilation function images obtained by two subjects using the CT ventilation function imaging method based on unsupervised learning proposed by the present invention (referred to as the unsupervised learning method) and the conventional CT ventilation function imaging method (referred to as the conventional method) are shown. From the results, it can be seen that the results obtained by the unsupervised learning method and the conventional method are consistent, which verifies the accuracy of the method proposed in this patent. However, the calculation time required for the unsupervised learning method is much less than the calculation time required for the conventional method.

[0055] Figure 4 The quantitative comparison of the time required for the unsupervised learning method and the conventional method to calculate the ventilation function image using the same computer hardware and test set is shown. For one subject, the unsupervised learning method only took 3 seconds to calculate the ventilation function image, while the conventional method took more than 1000 seconds to calculate, with a significant statistical difference between the two methods (P<0.001). Taking the evaluation of 100 subjects as an example, the unsupervised learning method only took 5 minutes to calculate the ventilation function image of all subjects, while the conventional method took more than 24 hours.

[0056] The specific embodiments of the present invention are described above. The specific embodiments described in the specification of the present invention are merely examples of the spirit of the present invention. Those skilled in the art of the present invention may make various modifications or additions to the described specific embodiments, or replace them in a similar manner, but they will not deviate from the spirit of the present invention or exceed the scope defined by the attached claims.

Claims

1. A CT ventilation function imaging method based on unsupervised learning, characterized in that: The following steps are involved: Step 1, collecting and preprocessing lung CT images of subjects in two different respiratory states, the lung CT images include CT1 images and CT2 images, and merging the preprocessed CT1 image and CT2 image of each subject into a dual-channel image; Step 2, constructing a CT ventilation function imaging network, the CT ventilation function imaging network includes an unsupervised registration module and a ventilation function imaging module, the unsupervised registration module generates an initial spatial transformation and an initial deformation image according to the dual-channel image, and the ventilation function imaging module generates a final total spatial transformation, a final deformation image and a ventilation function image according to the initial spatial transformation; Step 3: Construct the loss function L of the CT ventilation function imaging network. L=L sim +λ*L T Among them, L sim is the similarity measure between the CT2 image and the final deformed image, λ represents the weight of the constraint term of the spatial transformation, and L T is the constraint term of the final total space transformation; Step 4: Prepare a training set for the CT ventilation function imaging network, where dual-channel images of different subjects constitute samples in the training set; Step 5, performing unsupervised training on the CT ventilation function imaging network through the training set to obtain the final parameters of the CT ventilation function imaging network; Step 6: Acquire CT1 images and CT2 images of the new subject's lungs in two respiratory states, generate dual-channel images and input them into the CT ventilation function imaging network to obtain the ventilation function image of the new subject.

2. The CT ventilation function imaging method based on unsupervised learning according to claim 1, characterized in that: The pretreatment of step 1 comprises the following steps: Step 1.1, the resolution of all lung CT images is adjusted to the same resolution using an interpolation algorithm; Step 1.2, adjusting the CT value of the lung CT image to display only the lung window; Step 1.3: Merge the preprocessed CT1 image and CT2 image of each subject into a dual-channel image.

3. The CT ventilation function imaging method based on unsupervised learning according to claim 2, characterized in that: The unsupervised registration module of step 2 includes a downsampling stage, an upsampling stage, and a spatial deformation stage in sequence. The spatial deformation stage includes two convolutional layers. The first convolutional layer in the spatial deformation stage outputs an initial spatial transformation, and the second convolutional layer outputs an initial deformed image through a spatial deformation function according to the initial spatial transformation output by the first convolutional layer.

4. The CT ventilation function imaging method based on unsupervised learning according to claim 3, characterized in that: The unsupervised registration modules of step 2 are multiple and connected in series, and the input of the latter unsupervised registration module is the initial deformed image output by the previous unsupervised registration module.

5. The CT ventilation function imaging method based on unsupervised learning according to claim 4, characterized in that: The convolution kernels of the same unsupervised registration module are the same, and the convolution kernels of different unsupervised registration modules are different.

6. The CT ventilation function imaging method based on unsupervised learning according to claim 5, characterized in that: The ventilation function imaging module includes three convolutional layers, wherein the first convolutional layer merges the initial spatial transformations obtained by each unsupervised registration module to generate a final total spatial transformation; the second convolutional layer applies the final total spatial transformation to the CT1 image to generate a final deformed image; the third convolutional layer calculates the relative change of the CT value of each lung voxel in the CT2 image based on the CT2 image and the final deformed image to obtain a ventilation function image.

7. The CT ventilation function imaging method based on unsupervised learning according to claim 1, characterized in that: The similarity measure is the mean square error between the CT2 image and the final deformed image, or the normalized cross-correlation coefficient between the CT2 image and the final deformed image, or the similarity measure is the mutual information between the CT2 image and the final deformed image.

8. The CT ventilation function imaging method based on unsupervised learning according to claim 1, characterized in that: The constraint term L of the final total space transformation T Based on the following formula: Where p represents pixel, Ω represents image space, represents the gradient operator and T is the final total spatial transformation.

Images