An x-ray image reconstruction method based on a parallel three-channel convolutional neural network

Abstract

The application discloses an X-ray image reconstruction method based on a parallel three-channel convolutional neural network, which comprises the following steps: (1) collecting multiple sets of training data, each set of training data comprising three X-ray projection images of a security inspection machine and a corresponding CT sequence, and marking in the image data; (2) constructing a generator model and a discriminator model; (3) locking the parameters of the discriminator model, training the parameters of the generator model, and designing a loss function of the first stage; (4) locking the parameters of the generator model, training the parameters of the discriminator model, and designing a loss function of the second stage; (5) iteratively training steps (3)-(4) until the parameters of the generator and the discriminator model reach local optimization; (6) inputting three X-ray projection images to be reconstructed into the trained generator model, and outputting a three-dimensional CT sequence; and performing visualization by using a volume rendering algorithm. The model of the application has a fast inference speed, is easy to deploy, and has high objective indexes and high subjective similarity in the generated reconstruction effect.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer image generation, and particularly relates to an X-ray image reconstruction method based on a parallel three-channel convolutional neural network. BACKGROUND

[0002] The luggage inspection of the security inspection machine is the first link in the transportation scene, and aims to prevent the transportation of prohibited and potentially dangerous articles. In practical applications, observing the projection image generated by the X-ray scanner is the main way for security personnel to make judgments. However, the stacking of multiple objects can easily cause pixel compression and repetition on the two-dimensional image, especially when the medium of the top object is special, which hides the X-ray absorption amount of the underlying object, making it difficult for the naked eye to distinguish, and the staff needs to rely on experience to confirm repeatedly. Upgrading the two-dimensional image to restore it to a three-dimensional volume is an efficient discrimination method.

[0003] At present, three-dimensional reconstruction methods have been widely used in various fields, such as games, movies, autonomous driving, virtual humans, etc. Traditional multi-view geometry three-dimensional reconstruction algorithms mainly rely on parallax to recover the three-dimensional structure according to the different projection positions of the same three-dimensional point in different cameras.

[0004] For example, the Chinese patent document with publication number CN10319852A discloses a large-scale outdoor scene three-dimensional reconstruction method, which recovers the spatial information of the outdoor scene in the entire shooting range from the three-dimensional reconstruction results of local regions according to images collected by multiple viewpoints.

[0005] For example, the Chinese patent document with publication number CN112927348A discloses a high-resolution human three-dimensional reconstruction method based on multi-view RGBD cameras, which includes the following steps: preparing a data training set; extracting image features; sampling spatial points; calculating the position of each sampling point projected on the images collected by the multi-view cameras, and calculating the corresponding features; combining the spatial point depth information of the depth camera to calculate the psdf of each sampling point as the weight of the fusion of the different image feature values corresponding to the sampling point; using the psdf of each sampling point as the weight to fuse the feature values of the sampling points, and as the input of the MLP neural network; predicting the probability of each sampling point being on the model surface in space through the MLP neural network, thereby completing the geometric surface reconstruction of the human three-dimensional model, and using the same steps to complete the texture reconstruction of the human three-dimensional model surface, thereby obtaining a high-resolution human three-dimensional reconstruction model.

[0006] However, the calculation of the above-mentioned three-dimensional reconstruction method is easily affected by the selection of feature points and lighting, and the computational complexity is high. If the object 3D structure is to be accurately restored, a large number of input pictures are required, which is contrary to the actual scene where the security inspection machine generates 1-3 projection images that are mutually orthogonal.

[0007] With the development of deep learning, deep learning-based 3D reconstruction algorithms have high computational efficiency, can perform real-time inference and rendering, and have stronger rendering capabilities to solve 3D reconstruction problems in complex scenes. Therefore, it is urgent to design a deep learning-based 3D reconstruction method suitable for security scenarios. Summary of the Invention

[0008] To address the problems existing in current technologies, this invention provides an X-ray image reconstruction method based on a parallel three-channel convolutional neural network. The model boasts fast inference speed, ease of deployment, and generates reconstructions with both high objective metrics and high subjective similarity.

[0009] An X-ray image reconstruction method based on a parallel three-channel convolutional neural network can generate three-dimensional formats end-to-end in security scenarios, including the following steps:

[0010] (1) Collect multiple sets of training data in security scenarios. Each set of training data includes three X-ray projection images of industrial security inspection machines and corresponding CT sequence data. The camera angle and corresponding CT sequence are marked in the image data.

[0011] (2) Construct a generator model and a discriminator model; wherein, the generator model is used to expand the dimension of the three input two-dimensional images to generate a three-dimensional volume sequence; the discriminator model is used to take the three-dimensional volume sequence generated by the generator model as the prediction value input, and at the same time input the real CT sequence value to judge the correctness of the voxel generation;

[0012] (3) Lock the parameters of the discriminator model, train the parameters of the generator model using the training data, and design the loss function for the first stage; continuously reduce the learning rate during the training process until the error is less than a specific value and the model converges.

[0013] (4) Lock the generator model parameters, use the training data to train the discriminator model parameters, and design the loss function for the second stage; continuously reduce the learning rate during the training process until the error is less than a specific value and the model converges.

[0014] (5) Iterate through steps (3) to (4) for at least 50 epochs, using a generative adversarial training strategy, until the parameters of both the generator and discriminator models reach a local optimum.

[0015] (6) In practical applications, the three X-ray projection images to be reconstructed are input into the trained generator model, and the corresponding three-dimensional CT sequence is output. The volume rendering algorithm is used for visualization, and the three-dimensional information of the image is presented intuitively from multiple angles.

[0016] In step (1), in each set of training data, the three X-ray projection images correspond to the top view, front view, and side view, respectively, and the pixel size is the same and the length and width are equal.

[0017] In step (2), the generator model adopts three parallel Unet models corresponding to the X-ray projection images of the top view, the front view and the side view respectively; wherein the dense residual module is used as the encoder for down-sampling, and the 3DCNN module is used as the decoder for up-sampling, and the output of the i-th layer of the front view, side view and top view decoder is F out_i , P out_i and V out_i respectively.

[0018] The final output of the generator model is generated layer by layer by the decoder, wherein the input of the i-th layer of the generator model is the weighted average of the three outputs F out_i , P out_i , V out_i of the i-th layer of the decoder and the output D out_i-1 of the previous layer of the generator model, and the formula is:

[0019]

[0020] In the up-sampling stage of each parallel Unet model, the other two input images are introduced layer by layer, and the pixel after average pooling is used as the weight map for generation correction. Taking the front view decoder as an example, the specific operation is as follows:

[0021] The feature map output by the i-th layer decoder is F out_i (c,h,w), and each row and column feature value of the input of the next layer decoder is calculated as follows:

[0022] F tmp [c,h,:]=F out_i [c,h,:]·W p [h i ,w i ]

[0023] F in_i+1 [c,:,w]=F tmp [c,:,w]·W V [h i ,w i ]

[0024] Wherein, c, h, w are the channel number, row number and column number of the i-th layer feature map respectively, F out_i [c,h,:] is all feature values of a column of the i-th layer output feature map, F tmp [c,h,:] is all feature values of a column of the temporary feature map, F tmp [c,:,w] is all feature values of a row of the temporary feature map, F in_i+1 [c,:,w] is all feature values of a row of the i+1-th layer input feature map, and W p [hi ,w i ]With W V [h i ,w i ] is the weight map of the i-th layer, which is obtained by pooling and regularizing the other two input images, and the specific position of the multiplication factor in the weight map is determined according to the spatial relationship of the three views.

[0025] In step (2), the discriminator model adopts a Patch-GAN network.

[0026] In step (3), the loss function of the first stage adds a pixel-level and feature-level objective function to balance the subjective evaluation effect and objective index calculation of the generated model, and the specific formula is as follows:

[0027]

[0028] Wherein, l GAN (G) is the adversarial loss function, l pec is the perceptual loss function, l smooth is the L1-Smooth loss function, and alpha1, alpha2, alpha3 are weights, and the values are 0.2, 1 and 0.5.

[0029] In step (4), the loss function of the second stage adopts the LSGAN adversarial loss.

[0030] In step (6), the ray-casting algorithm is used for volume rendering.

[0031] Compared with the prior art, the present application has the following beneficial effects:

[0032] (1) The three-channel parallel model proposed in the present application can obtain more effective information during feature extraction compared with the two-channel or single-channel of other reconstruction algorithms, and the two additional images input are used as weight maps for correction, so that the generated new voxels are more reasonable.

[0033] (2) The model of the present application combines pixel-level loss function and feature-level loss function during training, and the reconstruction generation effect has high objective index and high subjective similarity.

[0034] (3) The model of the present application has fewer overall parameters, and compared with large generation networks, it has faster inference generation speed and is easy to deploy on ordinary terminal devices. DETAILED DESCRIPTION

[0035] Figure 1 is a flowchart of an X-ray image reconstruction method based on a parallel three-channel convolutional neural network according to the present application;

[0036] Figure 2 is a schematic diagram of a three-channel convolutional neural network of the generator model according to the present application;

[0037] Figure 3 This is a schematic diagram of the upsampling fusion decoder in the generator model of this invention;

[0038] Figure 4 This is a schematic diagram of the weight correction process in this invention;

[0039] Figure 5 This is a diagram showing the relationship between the weight correction method and the viewpoint in this invention. Detailed Implementation

[0040] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments described below are intended to facilitate the understanding of the present invention and do not constitute any limitation thereof.

[0041] like Figure 1 As shown, an X-ray image reconstruction method based on a parallel three-channel convolutional neural network can generate three-dimensional formats end-to-end in security scenarios, including the following steps:

[0042] Step 1: Collect multiple sets of training data in security scenarios. Each set of training data includes three X-ray projection images of the security inspection machine and the corresponding CT sequence data.

[0043] The three required 2D X-ray images should be orthogonal to each other, i.e., top-view, front-view, and side-view, and have the same pixel size and equal length and width. The CT sequence should consist of at least 128 images, and the shooting angle and corresponding CT sequence should be labeled in the image data for training.

[0044] Step 2: Design the generator model for the first stage, which is used to expand the dimensions of the three input two-dimensional images to generate a three-dimensional volume sequence.

[0045] like Figure 2 As shown, the generator model as a whole adopts three parallel Unet models. For each channel of the Unet model, downsampling uses a dense residual block as the encoder, and upsampling uses a 3DCNN module as the decoder. The output of the i-th layer is F. out_i ,P out_i V out_i .

[0046] like Figure 3 As shown, for the final output of the generator model, a new 3DCNN network is constructed to reconstruct voxels layer by layer, where the input of each layer is F. out_i ,P out_i V out_i and the output D of the previous layer out_i-1 The weighted average sum, i.e.:

[0047]

[0048] As Figure 4 shown, in the up-sampling stage of each parallel Unet model, the other two input images are introduced layer by layer, and the pixel after average pooling is used as a weight map to generate correction. The specific operation is as follows: the feature map output by the ith layer is F out_i (c,h,w), then each row and column of feature values of the input of the next layer decoder is calculated as follows:

[0049] F tmp [c,h,:]=F out_i [c,h,:]·W p [h i ,w i ]

[0050] F in_i+1 [c,:,w]=F tmp [c,:,w]·W V [h i ,w i ]

[0051] Where c, h, w are the channel number, row number, and column number of the ith layer feature map, respectively, F out_i [c,h,:] is the entire feature value of a column of the output feature map of the ith layer, F tmp [c,h,:] is the entire feature value of a column of the temporary feature map, F tmp [c,:,w] is the entire feature value of a row of the temporary feature map, F in_i+1 [c,:,w] is the entire feature value of a row of the input feature map of the ith+1 layer, W p [h i ,w i ] and W V [h i ,w i ] are the weight maps of the ith layer, which are obtained by pooling and regularization of the other two input images.

[0052] The entire element of a certain dimension of the feature map to be corrected is multiplied by the corresponding position factor in the weight map, that is, the correction of the feature value can be completed, as shown in Figure 4 The factor of the hth row and wth column in the side view angle weight map corrects all column features in the hth row of the wth channel of the front view angle encoder.

[0053] As Figure 5 shown, the dimension information of the corrected element and the specific coordinates of the correction weight factor are one-to-one corresponding in position according to the spatial characteristics of the actual three-view angle.

[0054] Step 3, the discriminator model of the second stage is designed, the discriminator model adopts the Patch-GAN network, the three-dimensional sequence generated in the first stage is taken as the prediction value and input, and the real value is input at the same time, so as to judge the correctness of the voxel generation.

[0055] Step 4, the discriminator model parameters are locked, the generator model parameters are trained, and the loss function of the first stage is designed. The pixel-level and feature-level objective functions are added to balance the subjective evaluation effect and objective index calculation of the model generation, and the learning rate is continuously reduced in the training process until the error is less than a certain value, and the model reaches convergence.

[0056] The loss function of the generator model is as follows:

[0057]

[0058] Wherein, l GAN (G) is the adversarial loss function, l pec is the perceptual loss function, l smooth is the L1-Smooth loss function, and α1, α2, α3 are weights, and the values are 0.2, 1 and 0.5.

[0059] Step 5, the generator model parameters are locked, the discriminator model parameters are trained, the loss function of the second stage is designed, the pixel-level LSGAN loss function is introduced, and the learning rate is continuously reduced in the training process until the error is less than a certain value, and the model reaches convergence.

[0060] Step 6, steps 4-5 are iteratively trained for at least 50 epochs, and the generative adversarial training strategy is adopted until the generator and discriminator model parameters reach local optimization.

[0061] Step 7, in the actual application process, the three X-ray projection images to be reconstructed are input into the trained generator network, and the corresponding three-dimensional CT sequence is output, the ray-casting algorithm is used for volume rendering, the three-dimensional sequence is visualized and output, and finally the three-dimensional information of the reconstructed multi-angle intuitive final presentation image is completed.

[0062] The above embodiments have described the technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only a specific embodiment of the present application and is not used to limit the present application. Any modification, supplement and equivalent replacement made within the principle range of the present application shall be included in the protection range of the present application.

Claims

1. A method for X-ray image reconstruction based on a parallel three-channel convolutional neural network, characterized in that, The method comprises the following steps: (1) Collecting a plurality of sets of training data under a security scene, each set of training data comprising three X-ray projection images of an industrial security inspection machine and corresponding CT sequence data, and marking the camera viewing angle and the corresponding CT sequence in the image data; (2) Constructing a generator model and a discriminator model; wherein the generator model is used to expand the dimension of the input three two-dimensional pictures to generate a three-dimensional body sequence; the discriminator model is used to input the three-dimensional body sequence generated by the generator model as a prediction value, and simultaneously input the real CT sequence value, so as to judge the correctness of the voxel generation; (3) Locking the parameters of the discriminator model, training the parameters of the generator model using the training data, and designing a first-stage loss function; the learning rate is continuously reduced during the training process until the error is less than a certain value, and the model reaches convergence; The generator model employs three parallel Unet models, corresponding to X-ray projection images from top-down, front-view, and side-view perspectives, respectively. Downsampling uses a dense residual module as the encoder, and upsampling uses a 3DCNN module as the decoder. The outputs of the decoders at the i-th layer, corresponding to the front-view, side-view, and top-view perspectives, are Fi, ... out_i P out_i and V out_i ; In the upsampling stage of each parallel Unet model, the other two input images are introduced layer by layer, and the pixel after average pooling is used as a weight map for generation correction. The specific operation is as follows: The feature map output by the i-th layer decoder is F out_i (c, h, w), then each row and column of feature values input to the next layer decoder is calculated as follows: F tmp [c,h,:] = F out_i [c,h,:] · W p [h i ,w i ] F in_i+1 [c,:,w] = F tmp [c,:,w] · W V [h i ,w i ] Wherein, c, h, w are respectively the channel number, row number, column number of the i-th layer feature map, F out_i [c, h, :] is all feature values of a column of the i-th layer output feature map, F tmp [c, h, :] is all feature values of a column of the temporary feature map, F tmp [c, :, w] is all feature values of a row of the temporary feature map, F in_i+1 [c, :, w] is all feature values of a row of the i+1-th layer input feature map, W p [h i ,w i ] and W V [h i ,w i ] are the i-th layer weight map, obtained by pooling and regularizing another two input images, the specific position of the multiplication factor in the weight map is determined according to the spatial relationship of the three views; (4) Locking the parameters of the generator model, training the parameters of the discriminator model using the training data, and designing a second-stage loss function; the learning rate is continuously reduced during the training process until the error is less than a certain value, and the model reaches convergence; (5) Iteratively training steps (3) to (4) for at least 50 epochs, using a generative adversarial training strategy until the parameters of the generator and the discriminator model reach local optimality; (6) In the actual application process, input the three X-ray projection images to be reconstructed into the trained generator model, and output the corresponding three-dimensional CT sequence; use a volume rendering algorithm for visualization, and intuitively present the three-dimensional information of the image from multiple angles.

2. The method of claim 1, wherein the parallel three-channel convolutional neural network-based X-ray image reconstruction method is characterized by, In step (1), in each set of training data, the three X-ray projection images correspond to the top view, front view and side view angles respectively, and the pixel size is consistent, with equal length and width. 3.The X-ray image reconstruction method based on parallel three-channel convolutional neural network according to claim 1, characterized in that, The final output of the generator model is generated layer by layer by the decoder, wherein the input of the i-th layer of the generator model is the three outputs F out_i , P out_i , V out_i of the i-th layer of the decoder and the weighted average of the output D out_i-1 of the previous layer of the generator model, and the formula is:

4. The parallel three-channel convolutional neural network based X-ray image reconstruction method of claim 1, wherein, In step (2), the discriminator model uses a Patch-GAN network.

5. The parallel three-channel convolutional neural network based X-ray image reconstruction method of claim 1, wherein, In step (3), the first-stage loss function is as follows: wherein, l GAN (G) is an adversarial loss function, l pec is a perception loss function, l smooth is an L1-Smooth loss function, and a1, a2, a3 are weights with values of 0.2, 1, 0.

5.

6. The parallel three-channel convolutional neural network based X-ray image reconstruction method of claim 1, wherein, In step (4), the second-stage loss function uses LSGAN adversarial loss.

7. The parallel three-channel convolutional neural network based X-ray image reconstruction method of claim 1, wherein, In step (6), the ray-casting algorithm is used for volume rendering.

Images