X-ray multi-contrast image fusion characterization method based on unsupervised feature learning

Abstract

The invention relates to an X-ray multi-contrast image fusion representation method based on unsupervised feature learning, and belongs to the technical field of deep learning and X-ray multi-contrast imaging. The method comprises the following steps: acquiring absorption, phase and dark field contrast computed tomography images of X-ray grating imaging; constructing a generative adversarial network model based on unsupervised feature learning; training a generative adversarial network model based on unsupervised feature learning; and fusing the absorption, phase and dark field contrast computed tomography images by using the trained model to obtain a fused image, and representing a detected object by using the fused image. Compared with an existing X-ray grating multi-contrast imaging technology, the X-ray grating differential phase contrast imaging method and device can well utilize the advantage that X-ray grating differential phase contrast imaging can generate three different contrast images at the same time, and fusion characterization is conducted on the three contrast images.

Description

Technical Field

[0001] The present invention belongs to the technical fields of deep learning and X-ray grating diffraction imaging, and particularly relates to an X-ray multi-contrast image fusion characterization method based on unsupervised feature learning. Background Art

[0002] In X-ray imaging methods, the X-ray grating interferometry imaging method can use an ordinary X-ray tube for imaging. In this imaging method, by utilizing the attenuation, phase shift, and small-angle scattering that occur when X-rays interact with matter, three contrast images of absorption, phase, and dark field are resolved based on the Talbot effect to characterize the object to be inspected. Among them, the absorption contrast image can better distinguish objects with different densities, but has poor contrast for low-density substances. The phase contrast image, on the other hand, can provide superior contrast for low-density substances. In addition, the dark field contrast can provide good imaging for small structures and the edges of structures.

[0003] In biomedical research, different contrasts of X-ray grating imaging can be used to image different tissues. For example, the absorption contrast can provide good imaging contrast for tissues such as bones and teeth; the phase contrast can provide good imaging for soft tissues; the dark field contrast has considerable potential in the clinical applications of blood vessel imaging and lung imaging. However, in biomedical research, the object to be inspected is usually not a single tissue, but an organism with multiple complex tissues. This leads to the need for repeated comparison when interpreting the images of the three contrasts, causing certain difficulties in image interpretation. Especially in intelligent interpretation technologies, the work of annotating the images of the three contrasts will be very complicated and error-prone.

[0004] Therefore, if the image features of the three contrasts are fused into one image for characterization, the reading efficiency can be greatly improved and an efficient and accurate input can be provided for intelligent interpretation technologies. Currently, no X-ray multi-contrast image fusion characterization method based on unsupervised feature learning has been found. Summary of the Invention

[0005] To solve the above technical problems, the present invention provides an X-ray multi-contrast image fusion characterization method based on unsupervised feature learning. The method includes a generative adversarial network. In the network, the feature enhancement module of the generator is composed of a densely connected network and a convolutional neural network based on the self-attention mechanism in parallel, which is responsible for extracting and enhancing the image features of three contrasts. The feature fusion module of the generator in the network is composed of a Unet network, which can fuse the extracted feature maps and finally output a fused image. This method can utilize the unique advantage of multi-contrast simultaneous imaging in X-ray grating imaging to fuse and characterize the three contrasts, providing a new X-ray grating imaging characterization method. The fused image contains the information of multiple contrast images in one image, providing more accurate and comprehensive image input for the intelligent interpretation technology applicable to X-ray grating multi-contrast imaging.

[0006] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] An X-ray multi-contrast image fusion characterization method based on unsupervised feature learning includes the following steps:

[0008] Step S1: Use an X-ray grating multi-contrast imaging system for imaging to obtain absorption, phase, and dark-field contrast computed tomography images, and construct a training dataset;

[0009] Step S2: Construct a conditional generative adversarial network model based on unsupervised feature learning. The network model includes a generator and a discriminator, which are used to fuse and characterize multi-contrast images. Among them, the generator includes a feature enhancement module and a feature fusion module. The feature enhancement module includes a densely connected network and a convolutional neural network based on the self-attention mechanism, and the feature fusion module includes a Unet network;

[0010] Step S3: Use the training dataset to train the constructed network model;

[0011] Step S4: Use the trained network model to fuse and characterize the absorption, phase, and dark-field contrast computed tomography images generated by the X-ray grating multi-contrast imaging system.

[0012] The beneficial effects of the present invention are as follows:

[0013] Compared with the existing X-ray grating phase-contrast imaging method, which generates three image contrasts to characterize the object to be inspected, the present invention fuses and characterizes the three contrast images, improving the information retrieval efficiency and interpretation efficiency, promoting the application of intelligent interpretation technology in X-ray grating multi-contrast imaging and improving its performance. Description of the Drawings

[0014] Figure 1Flowchart of the X-ray multi-contrast image fusion characterization method based on unsupervised feature learning of the present invention;

[0015] Figure 2 Schematic diagram of the X-ray grating multi-contrast imaging system for imaging;

[0016] Figure 3 Structural diagram of the conditional generative adversarial network model based on unsupervised feature learning provided by an embodiment of the present invention;

[0017] Figure 4 A set of absorption, phase, and dark-field contrast computed tomography images obtained using the grating multi-contrast imaging system, where (a) is the absorption contrast image, (b) is the phase contrast image, and (c) is the dark-field contrast image;

[0018] Figure 5 Fusion image obtained using the method of the present invention;

[0019] Figure 6 Comparison diagram of the effects of three-dimensional visualization of multi-layer absorption, phase, and dark-field contrast computed tomography images and the effects of three-dimensional visualization after fusion characterization using the method of the present invention, where (a) is the three-dimensional visualization effect of the absorption contrast image; (b) is the three-dimensional visualization effect of the phase contrast image; (c) is the three-dimensional visualization effect of the dark-field contrast image; (d) is the three-dimensional visualization effect of the fusion image obtained by implementing the method of the present invention.

[0020] Reference numerals:

[0021] 20. X-ray source, 21. Source grating, 22. Object under inspection, 23. Phase grating, 24. Absorption grating, 25. Detector. Detailed implementation manners

[0022] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation manners.

[0023] Figure 1 Flowchart of the X-ray multi-contrast image fusion characterization method based on unsupervised feature learning provided by an embodiment of the present invention; the present invention can fuse and characterize the three image contrasts obtained by X-ray grating multi-contrast imaging, providing a higher image reading efficiency.

[0024] Compared with existing image fusion methods, most are only applicable to the case of two source images. The method provided by the present invention can fuse and characterize three contrast images, and reflect the information carried in the three contrasts in one image, which is expected to promote the clinical application based on X-ray grating multi-contrast imaging, especially the application combined with intelligent interpretation technology. The specific steps of this method are as follows:

[0025] Step S1: Use an X-ray grating multi-contrast imaging system to perform imaging, obtain absorption, phase, and dark-field contrast computed tomography images, and construct a training dataset;

[0026] Figure 2 Schematic diagram of the X-ray grating multi-contrast imaging system described in step S1; as Figure 2 shown, the X-ray grating multi-contrast imaging system is based on a Talbot-Lau interferometer, as Figure 2 shown, and altogether includes six parts: an X-ray source 20, a source grating 21, an object to be inspected 22, a phase grating 23, an absorption grating 24, and a detector 25. The parameters of the Talbot-Lau interferometer should satisfy the following formulas (1)-(4):

[0027] (1)

[0028] (2)

[0029] (3)

[0030] (4)

[0031] Among them, represents the distance between the phase grating 23 and the absorption grating 24; is the magnification ratio, is the distance between the source grating 21 and the phase grating 23 directly; represents the th-order fractional Talbot distance; is the period of the phase grating 23, is the wavelength of the X-ray, is the period of the absorption grating 24, is the period of the source grating 21, is the width that allows the X-ray to pass through in each period of the source grating 21;

[0032] Step S2: Construct a conditional generative adversarial network model based on unsupervised feature learning. The network model includes a generator and a discriminator, and is used for fusing and characterizing multi-contrast images. Among them, the generator includes a feature enhancement module and a feature fusion module. The feature enhancement module includes a densely connected network and a convolutional neural network based on the self-attention mechanism. The feature fusion module includes a Unet network;

[0033] Figure 3It is the structural diagram of the conditional generative adversarial network model based on unsupervised feature learning described in step S2, including a generator and a discriminator. Among them, the generator includes a feature enhancement module and a feature fusion module. The generator uses computer tomography images with three contrasts as conditional inputs, enhances the structural features of absorption, phase, and dark-field contrast images using the feature enhancement module, and fuses the enhanced structural features of absorption, phase, and dark-field contrast images using the feature fusion module to obtain a fused image for characterizing the three contrasts. The discriminator discriminates the fused image from the input absorption, phase, and dark-field contrast images, and confronts the generator by judging the similarity between the fused image and the input absorption, phase, and dark-field contrast images, so as to achieve Nash equilibrium.

[0034] The feature enhancement module in the generator is characterized in that, as Figure 3 shown, it contains two parallel channels. The first channel adopts a densely connected network, including five convolutional layers with a size of 3×3, a stride of 1, and 40 channels connected in sequence, and the results of each convolutional layer are densely connected to enhance the local features of the image; the second channel adopts a convolutional neural network based on the self-attention mechanism, including two convolutional layers with a size of 3×3, a stride of 1, and output channels of 16 and 32 connected in sequence, then connecting to the self-attention module, and then two convolutional layers with a size of 3×3, a stride of 1, and output channels of 20 and 16 connected in sequence to enhance the global features of the input image. Among them, the self-attention module uses the self-attention mechanism, as Figure 3 shown, to obtain three sequences through three convolutional modules. Then, through the normalization exponential layer and the convolutional module, a feature map is obtained, and the feature map and the input map are aggregated to obtain an output feature map. The self-attention module can enable the network to have a larger receptive field, capture important features globally from the image, and mainly focus on enhancing the global features of the image. The feature fusion module in the generator is characterized in that, as Figure 3 shown, it adopts the architecture of a Unet. The input channels are all the feature maps generated by the feature enhancement module, and the output is a fused image. The architecture of the Unet combines the high-resolution features in the encoder and the low-resolution features in the decoder by using skip connections in the traditional encoder-decoder, retaining more information.

[0035] The discriminator, as Figure 3As shown, it includes three sub-discriminators: the first discriminator, the second discriminator, and the third discriminator, which are respectively used to discriminate the similarity between the fused image and the input absorption, phase, and dark-field contrast. Each sub-discriminator includes five convolutional layers with a size of 3×3, a stride of 2, and the number of output channels being 63, 96, 128, 256, and 512 connected in sequence. After the last convolutional layer, a fully connected layer is connected in sequence. Among them, each sub-discriminator is trained by the absorption, phase, dark-field contrast images and the fused image, and confronts with the generator to achieve the Nash equilibrium.

[0036] Step S3: Use the training data set to train the constructed network model;

[0037] The training process in Step S3 includes forward propagation, calculating loss, and backpropagation. Among them, forward propagation is completed based on the network model described in Step S2. Calculating loss includes generation loss, MSE loss, and gradient loss. Among them, the generation loss is calculated from the results of the discriminator in Step S2; the MSE loss is shown in Equation (5), and the gradient loss is shown in Equation (6). In the backpropagation process, the Adam algorithm is used to update the parameters of the network model;

[0038] (5)

[0039] (6)

[0040] Among them, respectively represent the pixel positions in the horizontal and vertical directions on the image; respectively represent the sizes in the horizontal and vertical directions of the image; respectively represent the absorption, phase, and dark-field contrast images input into the network; represents the fused image output by the network generator. abs, phase, and dark respectively represent the absorption, phase, and dark-field contrast images. represents calculating the gradient of the image, and its calculation method is shown in Equation (7):

[0041] (7)

[0042] Among them, img represents the image for gradient calculation, represents the convolution operation, K represents the convolution kernel, and the convolution kernel K is shown in Equation 8:

[0043] (8)

[0044] Step S4: Use the trained network model to perform fusion characterization on the absorption, phase, and dark-field contrast computed tomography images generated by the X-ray grating multi-contrast imaging system.

[0045] Embodiment

[0046] According to step S1, multiple sets of computed tomography images of absorption, phase, and dark-field contrast can be obtained as the training data set. The training set contains a total of 39 sets of images.

[0047] According to step S3, the training data set is input into the network model constructed in step S2 for training. The generator and discriminator are both iterated 100 generations, and the learning rate is set to 0.002.

[0048] In addition, to verify the effectiveness of the embodiments of the present invention, in addition to the training data set, another 195 sets of absorption, phase, and dark-field contrast images are obtained as the test data set. Figure 4 As one of the test data sets, it is the result of imaging a mouse using an X-ray grating multi-contrast imaging system. Among them, (a) is the absorption contrast image, (b) is the phase contrast image, and (c) is the dark-field contrast image.

[0049] As Figure 5 shown, it is one of the fusion results obtained by fusing the 195 sets of test data sets using the network model trained in step S3 (corresponding to the data in Figure 4 ).

[0050] To further demonstrate the effectiveness of the embodiments of the present invention, three-dimensional visualization is performed on the 195 sets of test data sets and the results, and the effect is as Figure 6 shown. Among them, (a) is the three-dimensional visualization effect of the absorption contrast image; (b) is the three-dimensional visualization effect of the phase contrast image; (c) is the three-dimensional visualization effect of the dark-field contrast image; (d) is the three-dimensional visualization effect of the fusion image output based on the method of the present invention. As can be seen from Figure 6 , the fusion image better shows the unique features of different contrasts and provides better contrast for different tissues.

[0051] In summary, according to the present invention, compared with the three contrasts obtained by traditional X-ray grating multi-contrast imaging, the present invention fuses and characterizes the three contrasts, and the obtained fusion image can simultaneously show the unique features of different contrasts, can provide better image contrast for multiple tissues at the same time, improve the reading efficiency, and provide support for the application of intelligent interpretation technology in the field of X-ray grating multi-contrast imaging.

[0052] The above specific embodiments further elaborate on the object, technical solution, and beneficial effects of the present invention. Those skilled in the art can clearly understand that for the convenience and brevity of description, only the division of the above network modules is used as an example. In practical applications, the above functions can be assigned to different network modules as needed, that is, the network structure can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the network described above can refer to the corresponding process in the foregoing method embodiments and will not be elaborated here again.

[0053] It should be understood that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some or all of the deep learning network structures and parameters; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An X-ray multi-contrast image fusion characterization method based on unsupervised feature learning, characterized in that It includes the following steps: Step S1: Use an X-ray grating multi-contrast imaging system to perform imaging, obtain absorption, phase, and dark-field contrast computed tomography images, and construct a training dataset; Step S2: Construct a conditional generative adversarial network model based on unsupervised feature learning. The network model includes a generator and a discriminator, which are used to perform fusion characterization on multi-contrast images. Among them, the generator includes a feature enhancement module and a feature fusion module. The feature enhancement module includes a densely connected network and a convolutional neural network based on the self-attention mechanism. The feature fusion module includes a Unet network; Step S3: Use the training dataset to train the constructed network model; Step S4: Use the trained network model to perform fusion characterization on the absorption, phase, and dark-field contrast computed tomography images generated by the X-ray grating multi-contrast imaging system.

2. The X-ray multi-contrast image fusion characterization method based on unsupervised feature learning according to claim 1, characterized in that The X-ray grating multi-contrast imaging system in step S1 includes an X-ray source, a source grating, a test object, a phase grating, an absorption grating, and a detector arranged in sequence along the optical path. The parameters of the imaging system satisfy: （1） （2） （3） （4） Among them, represents the distance between the phase grating and the absorption grating; is the magnification ratio, is the distance between the source grating and the phase grating; represents the th order fractional Talbot distance; is the period of the phase grating, is the wavelength of the X-ray, is the period of the absorption grating, is the period of the source grating, is the width that allows the X-ray to pass through in each period of the source grating.

3. The X-ray multi-contrast image fusion characterization method based on unsupervised feature learning according to claim 1, wherein In step S2, the generator uses the feature enhancement module to enhance the structural features of the absorption, phase, and dark-field contrast images, and uses the feature fusion module to fuse the structural features of the enhanced absorption, phase, and dark-field contrast images to obtain a fused image. The discriminator discriminates the fused image from the input absorption, phase, and dark-field contrast images, and confronts the generator by judging the similarity between the fused image and the input absorption, phase, and dark-field contrast images to reach the Nash equilibrium.

4. The X-ray multi-contrast image fusion characterization method based on unsupervised feature learning according to claim 3, characterized in that The feature enhancement module of the generator includes two parallel channels. The first channel uses a densely connected network, including five convolutional layers with a size of 3×3, a stride of 1, and an output channel number of 40 connected in sequence, which is used to focus on enhancing the local features of the input absorption, phase, and dark-field contrast images. The second channel uses a convolutional neural network based on the self-attention mechanism, including two convolutional layers with a size of 3×3, a stride of 1, and output channel numbers of 16 and 32 connected in sequence, then connected to a self-attention module, and then two convolutional layers with a size of 3×3, a stride of 1, and output channel numbers of 20 and 16 connected in sequence, which is used to focus on enhancing the global features of the input absorption, phase, and dark-field contrast images.

5. The X-ray multi-contrast image fusion characterization method based on unsupervised feature learning according to claim 3, wherein The feature fusion module of the generator includes a Unet network architecture. All the feature maps output by the feature enhancement module are input into the Unet network architecture to output a fused image.

6. The X-ray multi-contrast image fusion characterization method based on unsupervised feature learning according to claim 3, wherein The discriminator includes three sub-discriminators, which are respectively used to discriminate the similarity between the fused image and the input absorption, phase, and dark-field contrast images. Each sub-discriminator includes five convolutional layers with a size of 3×3, a stride of 2, and output channel numbers of 63, 96, 128, 256, and 512 connected in sequence. A fully connected layer is connected in sequence after the last convolutional layer. Among them, each sub-discriminator is trained by the absorption, phase, and dark-field contrast images and the fused image, and confronts the generator to reach the Nash equilibrium.

7. The X-ray multi-contrast image fusion characterization method based on unsupervised feature learning according to claim 1, characterized in that The training process in step S3 includes forward propagation, loss calculation, and backpropagation. Among them, forward propagation is completed based on the network model described in step S2. Loss calculation includes generation loss, MSE loss, and gradient loss. The generation loss is calculated from the results of the discriminator in step S2. The MSE loss is shown in formula (5), and the gradient loss is shown in formula (6). In the backpropagation process, the Adam algorithm is used to update the parameters of the network model: （5） （6） Among them, respectively represent the pixel positions in the horizontal and vertical directions on the image; respectively represent the sizes in the horizontal and vertical directions of the image; respectively represent the absorption, phase, and dark-field contrast images input into the network model; represents the fused image output by the generator, represents calculating the gradient of the image, and its calculation method is shown in formula (7): （7） Among them, img represents the image for gradient processing, represents the convolution operation, K represents the convolution kernel, and the convolution kernel K is shown in formula (8): （8）。

Images