X-ray multi-contrast image fusion representation method based on unsupervised feature learning

Abstract

The present invention relates to an X-ray multi-contrast image fusion characterization method based on unsupervised feature learning, and belongs to the field of deep learning and X-ray multi-contrast imaging technology. The method comprises: obtaining absorption, phase, and dark field contrast computed tomography images of X-ray grating imaging; constructing a generative adversarial network model based on unsupervised feature learning; training a generative adversarial network model based on unsupervised feature learning; using the trained model to fuse the absorption, phase, and dark field contrast computed tomography images to obtain a fused image, and using the fused image to characterize the inspected object. Compared with the existing X-ray grating multi-contrast imaging technology, the embodiment of the present invention can better utilize the advantage of X-ray grating differential phase contrast imaging that can produce three different contrast images at the same time, and perform fusion characterization on the three contrast images.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and X-ray grating diffraction imaging technology, and in particular relates to an X-ray multi-contrast image fusion characterization method based on unsupervised feature learning. Background Art

[0002] Among X-ray imaging methods, X-ray grating interferometry imaging can be used with ordinary X-ray tubes. This imaging method exploits the attenuation, phase shift, and small-angle scattering that occur when X-rays interact with matter, using the Talbot effect to generate three contrast images: absorption, phase, and dark-field. Absorption contrast images can effectively distinguish objects of varying densities, but offer poor contrast for low-density materials. Phase contrast images, on the other hand, provide superior contrast for low-density materials. Furthermore, dark-field contrast allows for better imaging of tiny structures and their edges.

[0003] In biomedical research, the different contrast levels of X-ray grating imaging can be used to image different tissues. For example, absorption contrast can provide better imaging contrast for tissues such as bones and teeth; phase contrast can better image soft tissues; and dark-field contrast has considerable potential in clinical applications of vascular imaging and lung imaging. However, in biomedical research, the subjects are usually not single tissues, but organisms with multiple complex tissues. This requires repeated comparison when interpreting images with three contrast levels, which creates certain difficulties in image interpretation. In particular, in intelligent interpretation technology, the task of labeling images with three contrast levels is very complicated and prone to errors.

[0004] Therefore, fusing the image features of the three contrasts into a single image for representation can greatly improve image reading efficiency and provide an efficient and accurate input for intelligent interpretation technology. Currently, no X-ray multi-contrast image fusion representation method based on unsupervised feature learning has been found. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention provides an X-ray multi-contrast image fusion characterization method based on unsupervised feature learning. The method includes a generative adversarial network, in which the feature enhancement module of the generator of the network is composed of a densely connected network and a convolutional neural network based on a self-attention mechanism in parallel, which is responsible for extracting and enhancing the image features of the three contrasts. The feature fusion module of the generator of the network is composed of a Unet network, which can fuse the extracted feature maps and finally output a fused image. The method can fuse the three contrasts by taking advantage of the unique multi-contrast simultaneous imaging of X-ray grating imaging, and provides a new X-ray grating imaging characterization method. The fused image contains information of multiple contrast images in one image, providing more accurate and comprehensive image input for the intelligent interpretation technology suitable for X-ray grating multi-contrast imaging.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] The X-ray multi-contrast image fusion representation method based on unsupervised feature learning includes the following steps:

[0008] Step S1, using an X-ray grating multi-contrast imaging system to perform imaging, obtain absorption, phase, and dark field contrast computed tomography images, and construct a training data set;

[0009] Step S2: constructing a conditional generative adversarial network model based on unsupervised feature learning, wherein the network model includes a generator and a discriminator, and is used to perform fusion representation on the multi-contrast image, wherein the generator includes a feature enhancement module and a feature fusion module, the feature enhancement module includes a densely connected network and a convolutional neural network based on a self-attention mechanism, and the feature fusion module includes a Unet network;

[0010] Step S3: using the training data set to train the constructed network model;

[0011] Step S4: Use the trained network model to perform fusion representation on the absorption, phase, and dark field contrast computed tomography images generated by the X-ray grating multi-contrast imaging system.

[0012] The beneficial effects of the present invention are:

[0013] Compared with the existing X-ray grating phase contrast imaging method, which generates three image contrasts to characterize the object under inspection, the present invention fuses the three contrast images for representation, improves the information retrieval efficiency and interpretation efficiency, promotes the application of intelligent interpretation technology in X-ray grating multi-contrast imaging and improves its performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1This is a flow chart of the X-ray multi-contrast image fusion characterization method based on unsupervised feature learning of the present invention;

[0015] Figure 2 This is a schematic diagram of the X-ray grating multi-contrast imaging system;

[0016] Figure 3 A diagram showing the structure of a conditional generative adversarial network model based on unsupervised feature learning provided by an embodiment of the present invention;

[0017] Figure 4 A set of absorption, phase, and dark-field contrast computed tomography images acquired using a grating multi-contrast imaging system, where (a) is an absorption contrast image, (b) is a phase contrast image, and (c) is a dark-field contrast image.

[0018] Figure 5 is a fused image obtained using the method of the present invention;

[0019] Figure 6 The figures compare the effects of three-dimensional visualization of multi-layer absorption, phase, and dark-field contrast computed tomography images and the effects of three-dimensional visualization after fusion characterization using the method of the present invention, wherein (a) is the three-dimensional visualization effect of the absorption contrast image; (b) is the three-dimensional visualization effect of the phase contrast image; (c) is the three-dimensional visualization effect of the dark-field contrast image; and (d) is the three-dimensional visualization effect of the fused image obtained by implementing the method of the present invention.

[0020] Reference numerals:

[0021] 20. X-ray source, 21. Source grating, 22. Object under inspection, 23. Phase grating, 24. Absorption grating, 25. Detector. DETAILED DESCRIPTION

[0022] The present invention is further described below with reference to the accompanying drawings and specific embodiments.

[0023] Figure 1 Flowchart of the X-ray multi-contrast image fusion characterization method based on unsupervised feature learning provided in an embodiment of the present invention; the present invention can fuse and characterize the three image contrasts obtained by X-ray grating multi-contrast imaging, providing higher image reading efficiency.

[0024] Compared to existing image fusion methods, most of which are only applicable to two source images, the method provided by the present invention can fuse and represent three contrast images, integrating the information carried by the three contrasts into a single image. This is expected to promote the clinical application of X-ray grating multi-contrast imaging, especially when combined with intelligent interpretation technology. The specific steps of this method are as follows:

[0025] Step S1, using an X-ray grating multi-contrast imaging system to perform imaging, obtain absorption, phase, and dark field contrast computed tomography images, and construct a training data set;

[0026] Figure 2 is a schematic diagram of the X-ray grating multi-contrast imaging system described in step S1; Figure 2 As shown, the X-ray grating multi-contrast imaging system is based on Talbot-Lau interferometer, as shown in Figure 2 As shown, it includes 6 parts in total: X-ray source 20, source grating 21, object under test 22, phase grating 23, absorption grating 24, and detector 25. The parameters of the Talbot-Lau interferometer should satisfy the following formulas (1)-(4):

[0027] (1)

[0028] (2)

[0029] (3)

[0030] (4)

[0031] in, represents the distance between the phase grating 23 and the absorption grating 24; is the magnification ratio, is the distance between the source grating 21 and the phase grating 23; Indicates the Fractional Talbot distance; is the period of the phase grating 23, is the wavelength of X-rays, is the period of the absorption grating 24, is the period of the source grating 21, is the width of the source grating 21 that allows X-rays to pass through in each period;

[0032] Step S2: constructing a conditional generative adversarial network model based on unsupervised feature learning, wherein the network model includes a generator and a discriminator, and is used to perform fusion representation on the multi-contrast image, wherein the generator includes a feature enhancement module and a feature fusion module, the feature enhancement module includes a densely connected network and a convolutional neural network based on a self-attention mechanism, and the feature fusion module includes a Unet network;

[0033] Figure 3This is a structural diagram of the conditional generative adversarial network model based on unsupervised feature learning described in step S2, which includes a generator and a discriminator. The generator includes a feature enhancement module and a feature fusion module. The generator uses three contrast computed tomography images as conditional inputs, uses the feature enhancement module to enhance the structural features of the absorption, phase, and dark field contrast images, and uses the feature fusion module to fuse the enhanced structural features of the absorption, phase, and dark field contrast images to obtain a fused image to characterize the three contrasts. The discriminator discriminates the fused image with the input absorption, phase, and dark field contrast images, and competes with the generator by determining the degree of similarity between the fused image and the input absorption, phase, and dark field contrast images, thereby achieving a Nash equilibrium.

[0034] The feature enhancement module in the generator is characterized by: Figure 3 As shown, it contains two parallel channels. The first channel uses a densely connected network, including sequentially connecting five convolutional layers with a size of 3×3, a stride of 1, and a number of channels of 40, and densely connecting the results of each convolutional layer to focus on the enhancement of local features of the image; the second channel uses a convolutional neural network based on the self-attention mechanism, including sequentially connecting two convolutional layers with a size of 3×3, a stride of 1, and output channels of 16 and 32 respectively, connecting the self-attention module, and then sequentially connecting two convolutional layers with a size of 3×3, a stride of 1, and output channels of 20 and 16 respectively, to focus on the enhancement of the global features of the input image. Among them, the self-attention module uses the self-attention mechanism, as shown in Figure 3 As shown, three sequences are obtained through three convolution modules. Then, a feature map is obtained through a normalized exponential layer and a convolution module. The feature map and the input map are aggregated to obtain an output feature map. The self-attention module can enable the network to have a larger receptive field, capture important features from the global image, and focus on enhancing the global features of the image. The feature fusion module in the generator is characterized by Figure 3 As shown, a Unet architecture is used. The input channels are all feature maps generated by the feature enhancement module, and the output is a fused image. The Unet architecture uses skip connections in traditional encoder-decoders to combine high-resolution features in the encoder and low-resolution features in the decoder, preserving more information.

[0035] The discriminator, such as Figure 3As shown in the figure, it contains three sub-discriminators: the first discriminator, the second discriminator, and the third discriminator, which are used to determine the similarity between the fused image and the input in terms of absorption, phase, and dark field contrast, respectively. Each sub-discriminator consists of five sequentially connected convolutional layers with a size of 3×3, a stride of 2, and output channels of 63, 96, 128, 256, and 512, respectively. The last convolutional layer is followed by a fully connected layer. Each sub-discriminator is trained using absorption, phase, dark field contrast images, and the fused image, and competes with the generator to achieve a Nash equilibrium.

[0036] Step S3: using the training data set to train the constructed network model;

[0037] The training process described in step S3 includes forward propagation, loss calculation, and back propagation. The forward propagation is completed based on the network model described in step S2. The loss calculation includes generation loss, MSE loss, and gradient loss. The generation loss is calculated based on the result of the discriminator in step S2. The MSE loss is shown in formula (5), and the gradient loss is shown in formula (6). The back propagation process uses the Adam algorithm to update the network model parameters.

[0038] (5)

[0039] (6)

[0040] in, Represent the pixel positions in the horizontal and vertical directions of the image respectively; Represents the horizontal and vertical dimensions of the image respectively; Represent the absorption, phase, and dark field contrast images input to the network respectively; Represents the fused image output by the network generator, abs, phase, and dark represent absorption, phase, and dark field contrast images, respectively. Represents the gradient of the image, and its calculation method is shown in formula (7):

[0041] (7)

[0042] Among them, img represents the image to be gradient, Represents the convolution operation, K represents the convolution kernel, and the convolution kernel K is shown in Formula 8:

[0043] (8)

[0044] Step S4: Use the trained network model to perform fusion representation on the absorption, phase, and dark field contrast computed tomography images generated by the X-ray grating multi-contrast imaging system.

[0045] Example

[0046] According to step S1, multiple sets of absorption, phase, and dark field contrast computed tomography images can be obtained as training data sets, which contain a total of 39 sets of images.

[0047] According to step S3, the training data set is input into the network model constructed in step S2 for training. The generator and the discriminator are iterated for 100 generations, and the learning rate is set to 0.002.

[0048] Furthermore, in order to verify the effectiveness of the embodiment of the present invention, in addition to the training data set, 195 sets of absorption, phase, and dark field contrast images were obtained as test data sets. Figure 4 This is one of the test datasets, and is the result of imaging a mouse using an X-ray grating multi-contrast imaging system. (a) is an absorption contrast image, (b) is a phase contrast image, and (c) is a dark field contrast image.

[0049] like Figure 5 As shown in FIG, one of the fusion results (corresponding to Figure 4 data in ).

[0050] To further demonstrate the effectiveness of the embodiment of the present invention, the 195 test data sets and results were visualized in three dimensions, and the results are as follows: Figure 6 As shown. Among them, (a) is the three-dimensional visualization effect of the absorption contrast image; (b) is the three-dimensional visualization effect of the phase contrast image; (c) is the three-dimensional visualization effect of the dark field contrast image; (d) is the three-dimensional visualization effect of the fusion image output based on the method of the present invention. Figure 6 As can be seen from the figure, the fused image better displays the unique features of different contrasts and provides better contrast for different tissues.

[0051] In summary, according to the present invention, compared with the three contrasts obtained by traditional X-ray grating multi-contrast imaging, the present invention fuses the three contrasts to represent them. The obtained fused image can simultaneously show the unique features of different contrasts, can provide better image contrast for multiple tissues at the same time, improve image reading efficiency, and provide support for the application of intelligent interpretation technology in the field of X-ray grating multi-contrast imaging.

[0052] The above specific embodiments further illustrate the purpose, technical solutions, and beneficial effects of the present invention. Those skilled in the art will clearly understand that for the sake of convenience and brevity, only the division of the above-mentioned network modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different network modules as needed, that is, the network structure can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the network described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0053] It should be understood that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some or all of the deep learning network structures and parameters therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An X-ray multi-contrast image fusion representation method based on unsupervised feature learning, characterized by: The steps include: Step S1, using an X-ray grating multi-contrast imaging system to perform imaging, obtain absorption, phase, and dark field contrast computed tomography images, and construct a training data set; Step S2: constructing a conditional generative adversarial network model based on unsupervised feature learning, wherein the network model includes a generator and a discriminator, and is used to fuse and represent multiple contrast images, wherein the generator includes a feature enhancement module and a feature fusion module, wherein the feature enhancement module includes a parallel dense connection network and a convolutional neural network based on a self-attention mechanism, which are respectively used to enhance local features and global features, and the feature fusion module includes a Unet network, which is used to fuse the enhanced features to obtain a fused image; the discriminator includes three sub-discriminators, which are respectively used to discriminate the similarity between the fused image and the input absorption, phase, and dark field contrast images; Step S3: using the training data set to train the constructed network model; Step S4: Use the trained network model to perform fusion representation on the absorption, phase, and dark field contrast computed tomography images generated by the X-ray grating multi-contrast imaging system.

2. The X-ray multi-contrast image fusion representation method based on unsupervised feature learning according to claim 1, characterized in that: The X-ray grating multi-contrast imaging system in step S1 includes an X-ray source, a source grating, an object to be inspected, a phase grating, an absorption grating, and a detector arranged in sequence along an optical path. The parameters of the imaging system satisfy the following requirements: （1） （2） （3） （4） in, represents the distance between the phase grating and the absorption grating; is the magnification ratio, is the distance between the source grating and the phase grating; Indicates the Fractional Talbot distance; is the period of the phase grating, is the wavelength of X-rays, is the period of the absorption grating, is the period of the source grating, is the width of the source grating that allows X-rays to pass through in each period.

3. The X-ray multi-contrast image fusion representation method based on unsupervised feature learning according to claim 1, characterized in that: In step S2, the generator uses a feature enhancement module to enhance the structural features of the absorption, phase, and dark field contrast images, and uses a feature fusion module to fuse the enhanced structural features of the absorption, phase, and dark field contrast images to obtain a fused image. The discriminator discriminates the fused image with the input absorption, phase, and dark field contrast images, and competes with the generator by judging the degree of similarity between the fused image and the input absorption, phase, and dark field contrast images to achieve a Nash equilibrium.

4. The X-ray multi-contrast image fusion representation method based on unsupervised feature learning according to claim 3, characterized in that: The feature enhancement module of the generator includes two parallel channels. The first channel adopts a densely connected network, including sequentially connecting five convolutional layers with a size of 3×3, a step size of 1, and an output channel number of 40, which is used to focus on the enhancement of the local features of the input absorption, phase, and dark field contrast images; the second channel adopts a convolutional neural network based on the self-attention mechanism, including sequentially connecting two convolutional layers with a size of 3×3, a step size of 1, and output channels of 16 and 32 respectively, connecting the self-attention module, and then sequentially connecting two convolutional layers with a size of 3×3, a step size of 1, and output channels of 20 and 16 respectively, which is used to focus on the enhancement of the global features of the input absorption, phase, and dark field contrast images.

5. The X-ray multi-contrast image fusion representation method based on unsupervised feature learning according to claim 3, characterized in that: The feature fusion module of the generator includes a Unet network architecture, which inputs all feature maps output by the feature enhancement module into the Unet network architecture and outputs a fused image.

6. The X-ray multi-contrast image fusion representation method based on unsupervised feature learning according to claim 3, characterized in that: Each sub-discriminator includes five sequentially connected convolutional layers with a size of 3×3, a stride of 2, and output channels of 63, 96, 128, 256, and 512 respectively. A fully connected layer is sequentially connected after the last convolutional layer. Each sub-discriminator is trained by absorption, phase, dark field contrast images and fusion images, and competes with the generator to achieve Nash equilibrium.

7. The X-ray multi-contrast image fusion representation method based on unsupervised feature learning according to claim 1, characterized in that: The training process in step S3 includes forward propagation, loss calculation, and back propagation. The forward propagation is completed based on the network model in step S2. The loss calculation includes generation loss, MSE loss, and gradient loss. The generation loss is calculated by the result of the discriminator in step S2. The MSE loss is shown in formula (5), and the gradient loss is shown in formula (6). The back propagation process uses the Adam algorithm to update the network model parameters: （5） （6） in, Represent the pixel positions in the horizontal and vertical directions of the image respectively; Represents the horizontal and vertical dimensions of the image respectively; Represent the absorption, phase and dark field contrast images input to the network model respectively; represents the fused image output by the generator, Represents the calculation of the gradient of the image, and its calculation method is shown in formula (7): （7） Where img represents the image to be gradiented, * represents the convolution operation, K represents the convolution kernel, and the convolution kernel K is shown in formula (8): （8）。

Images