A gan-based three-dimensional reconstruction method for citrus x-ray images

Abstract

The application discloses a kind of based on GAN's citrus X-ray image three-dimensional reconstruction method, it is related to fruit sign extraction field, to achieve the above purpose, the technical scheme S1 of the application, obtains citrus X-ray picture, constructs picture two-dimensional structure data set, and uses CT reconstruction corresponding citrus three-dimensional structure;S2, two-dimensional and three-dimensional data are respectively preprocessed;S3, construct the GAN network model of 3D reconstruction: respectively construct 3D generator and 3D discriminator;S4, based on network model training citrus three-dimensional reconstruction model in step 2;S5, input the X-ray picture of the citrus to be reconstructed into the citrus three-dimensional reconstruction model obtained by training, output the 3D structure of reconstruction.Both can use only one X-ray image, quickly realize reconstruction, and also can utilize multiple images to carry out citrus three-dimensional reconstruction, obtain higher precision reconstruction model.

Description

TECHNICAL FIELD

[0001] The application relates to a GAN-based three-dimensional reconstruction method for citrus X-ray images, and mainly relates to the field of fruit sign extraction. BACKGROUND

[0002] Currently, three-dimensional data representing the inside of a fruit is mainly obtained through CT technology reconstruction, but the application of the method requires at least hundreds of X-ray images taken by rotation, and the reconstruction process is complex, resulting in long time consumption of fruit three-dimensional model reconstruction, which cannot be applied to real-time scenes. SUMMARY

[0003] In view of the above problems of the prior art, the application provides a GAN-based three-dimensional reconstruction method for citrus X-ray images, which can quickly realize reconstruction using only one X-ray image, and can also use multiple images to realize three-dimensional reconstruction of the citrus to obtain a higher-precision reconstruction model.

[0004] To achieve the above purpose, the technical scheme of the application is as follows:

[0005] S1, obtaining a citrus X-ray image, constructing a two-dimensional structure data set of the image, and using CT reconstruction to obtain a corresponding three-dimensional structure of the citrus;

[0006] S2, preprocessing the two-dimensional and three-dimensional data respectively;

[0007] S3, constructing a GAN network model for 3D reconstruction: constructing a 3D generator and a 3D discriminator respectively;

[0008] S4, training a three-dimensional reconstruction model of the citrus based on the network model in step 2;

[0009] S5, inputting the X-ray image of the citrus to be reconstructed into the three-dimensional reconstruction model of the citrus obtained by training, and outputting the reconstructed 3D structure.

[0010] Preferably, in S1, the data set is constructed: X-ray projection images of the citrus at different angles are obtained to form a two-dimensional image data set; and a three-dimensional structure of each citrus is reconstructed using a CT reconstruction algorithm to construct a three-dimensional structure data set containing the X-ray image of the citrus.

[0011] Preferably, in S2, the preprocessing includes cropping and scaling of the X-ray film, format conversion, down-sampling, window width and window position adjustment of the CT data, so that the X-ray image of the citrus and the corresponding three-dimensional structure of the citrus have the same perspective projection; and the cropped image is scaled to make the size of all the cropped X-ray images of the citrus consistent.

[0012] Preferably, in S3, the 3DGAN model comprises:

[0013] The game process is expressed by the following formula:

[0014]

[0015] where z is sampled from the noise distribution. Since we want to learn the nonlinear mapping from x-rays to CT, the generated CT volume should be consistent with the semantic information provided by the input x-rays;

[0016] The GAN discriminator loss is defined in this patent as:

[0017]

[0018] The generator loss is defined as:

[0019]

[0020] where x denotes the x-ray plane of the orange, and y is the corresponding CT volume.

[0021] The reconstruction loss is defined as:

[0022]

[0023] The projection loss is:

[0024] L pl = E x,y ||P-P(G(x))||

[0025] Given the definitions of the adversarial loss, reconstruction loss, and projection loss, our final objective function is expressed as:

[0026]

[0027]

[0028] where λ1, λ2, and λ3 control the relative importance of different loss terms. In this patent, λ1=0.1, λ2=λ3=10.

[0029] Preferably, wherein S3, in the encoder area, the input orange x-ray picture first passes through a Basic2d module composed of Conv2d-BN-RReLU in series in the generator. Then it is encoded by three Blocks, each of which is composed of four RReLU-BN-Conv2d in series, wherein the second Conv2d filter size is 1x1, which can reduce the number of features under the premise of effective coding, and access an identical Basic2d module to realize subsequent coding.

[0030] Preferably, the decoder structure of the generator is composed of De-Basic3d and Basic3d in an alternating series, wherein the De-Basic3d module comprises Deconv3d-BN-RReLU, and the Basic3d module comprises Conv3d-BN-RReLU;

[0031] In the generator, the mapping of two-dimensional data to three-dimensional data is completed through a dimension increasing operation. Specifically, the two-dimensional data obtained after encoding is first subjected to a max-pooling layer, and then the data dimension is reconstructed to three-dimensional using a view function. A shortcut connection composed of Basic2d-Expand-Basic3d is introduced after each level of encoding operation to copy and pass the information of different levels along the third dimension to the decoder.

[0032] Preferably, wherein S4, the citrus X-ray pictures in the data set are input into the 3D generator one by one, and the corresponding three-dimensional structure of the citrus is generated after the encoding and decoding operation of S3; then, the discriminator discriminates the probability that the generated three-dimensional structure is true and returns the result to the generator; the generator performs self-optimization according to the received feedback to generate a three-dimensional structure of the citrus with higher precision; then, the discriminator discriminates the newly generated three-dimensional structure of the citrus with higher fraudulence, and performs self-optimization according to the discrimination accuracy; finally, after alternating optimization training, the 3D generator and the 3D discriminator achieve Nash equilibrium, and the model can generate a high-precision three-dimensional CT structure of the citrus.

[0033] Preferably, wherein S3, the 3D generator, in the encoder region, first inputs a pair of orthogonal images into a three-dimensional data space, saves the two-dimensional data orthogonality in the space, and sets the voxel values of the remaining positions to 0 to construct an orthogonal three-dimensional model of the citrus; the input orthogonal three-dimensional model of the citrus is encoded by three Block3ds in the generator, and each Block3d is connected in series with four RReLU-BN-Conv3ds, wherein the second Conv3d has a filter size of 1x1x1, which can reduce the number of features under the premise of effective encoding.

[0034] The decoder structure of the generator is composed of De-Basic3d and Basic3d in an alternating series, wherein the De-Basic3d module comprises Deconv3d-BN-RReLU, and the Basic3d module comprises Conv3d-BN-RReLU.

[0035] The input is an orthogonal three-dimensional model of the citrus, and the output is a reconstructed three-dimensional model of the citrus after encoding and decoding processing. A shortcut connection composed of Basic3d is introduced after each level of encoding operation to pass the information of different levels to the decoder. When the number of orthogonal image groups is greater than 1, the three-dimensional model of the citrus of each group of orthogonal images is output, and the weight of each three-dimensional model is calculated according to the total loss, as follows:

[0036]

[0037] The final citrus three-dimensional model is obtained by weighted average processing of the voxel values of each citrus reconstructed three-dimensional model, as follows:

[0038]

[0039] Wherein, G * is the total loss of the citrus reconstructed three-dimensional model generator end, and sigma is the voxel value of the citrus reconstructed three-dimensional model;

[0040] The 3D discriminator is composed of three Conv3d-BN-RReLU structures and a single Conv3d in series.

[0041] Preferably, it comprises a conveyor belt, a collection chamber and a fruit holder, the collection chamber is arranged on the conveyor belt, the collection chamber is provided with a feeding port and a discharging port corresponding to the conveyor belt, the collection chamber is provided with a CCD camera, a ray source and a detector, the conveyor belt is spaced apart and provided with a plurality of fruit holders, the fruit holder is rotationally matched with the conveyor belt, and the fruit holder is driven by a motor.

[0042] The technical principles and beneficial effects of the present application are as follows:

[0043] The present application can use only one X-ray image to complete fruit three-dimensional reconstruction based on the self-built GAN three-dimensional reconstruction model, greatly simplifying the reconstruction process and shortening the reconstruction time, and can also use multiple images for citrus three-dimensional reconstruction, although the time is longer, but the model accuracy obtained by reconstruction is higher. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed in the embodiment description will be briefly introduced below, and obviously, only three drawings of the present application are described below, and other drawings can also be obtained by those skilled in the art without creative labor.

[0045] Figure 1 It is a schematic diagram of the 3D generator of the encoder-decoder structure of embodiment 1 of the present application.

[0046] Figure 2 It is a citrus three-dimensional reconstruction network structure based on multiple sets of orthogonal X-ray images of embodiment 2 of the present application.

[0047] Figure 3 It is a schematic diagram of the object table.

[0048] Among them, the conveyor belt 1, the fruit holder 2 and the collection chamber 3. DETAILED DESCRIPTION

[0049] The technical solutions in the present application will be described clearly and completely in combination with the drawings. Obviously, the described embodiments are only the preferred embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative work are within the protection scope of the present application.

[0050] Embodiment 1

[0051] A GAN-based X-ray three-dimensional reconstruction method for citrus, the present embodiment is used for three-dimensional reconstruction of the internal structure of citrus using a single X-ray image, the method comprising the following steps:

[0052] S1, obtaining citrus X-ray pictures, constructing a picture dataset, and using CT reconstruction to reconstruct the corresponding three-dimensional structure of the citrus.

[0053] Construction of the dataset: place the citrus to be tested on the sample stage, rotate the stage, and the X-ray source emits X-rays to realize X-ray projection image acquisition at different angles. Finally, use the obtained citrus X-ray pictures to form a two-dimensional picture dataset. At the same time, use the traditional CT reconstruction algorithm to reconstruct the three-dimensional structure of each citrus, and construct a three-dimensional structure dataset containing the corresponding three-dimensional structure of the citrus X-ray pictures.

[0054] S2, pre-process the two-dimensional and three-dimensional data respectively: including cropping and scaling of the X-ray film, format conversion, down-sampling, window width and window level adjustment of the CT data, etc. Ensure that the citrus X-ray pictures and the corresponding three-dimensional structure of the citrus have the same perspective projection.

[0055] Edge cropping is performed on the pictures in the picture dataset to ensure that the citrus image in the citrus X-ray picture is located in the middle position of the picture, so as to ensure that it has the same perspective projection as the corresponding three-dimensional structure of the citrus. Then, the cropped pictures are scaled for preprocessing (size transformation) to ensure that the sizes of all cropped citrus X-ray pictures are consistent.

[0056] S3, constructing a GAN network model for 3D reconstruction: constructing a 3D generator and a 3D discriminator respectively. The 3D generator is an encoder-decoder network, which aims to learn and realize the mapping from 2D X-ray to 3D CT structure by applying full connection layers. The 3D discriminator is a network architecture containing multiple Conv3d-Norm-Relu modules, which has the ability to distinguish true and false three-dimensional structures.

[0057] The 3DGAN model in S3 includes:

[0058] The game process is expressed by the following formula:

[0059]

[0060] where z is sampled from a noise distribution. Since we want to learn a nonlinear mapping from X-rays to CT, the generated CT volume should be consistent with the semantic information provided by the input X-rays.

[0061] The GAN discriminator loss described in this patent is defined as:

[0062]

[0063] The generator loss is defined as:

[0064]

[0065] Here, x refers to the citrus X-ray plane and y is the corresponding CT volume.

[0066] The reconstruction loss is defined as:

[0067]

[0068] The projection loss is:

[0069] L pl =E x,y ||pp(G(x))||

[0070] Given the definitions of adversarial loss, reconstruction loss, and projection loss, our final objective function is expressed as:

[0071]

[0072]

[0073] Among them, λ1, λ2, and λ3 control the relative importance of different loss terms. In this patent, λ1 is set to 0.1, λ2 = λ3 = 10.

[0074] like Figure 1 As shown in the figure, this patent constructs a 3D generator with an encoder-decoder structure. In the encoder area, the input citrus X-ray image first passes through a Basic2d module composed of a series of Conv2d, BN, and RReLU. The encoding is then performed by three blocks, each of which is a series of four RReLU-BN-Conv2d blocks. The second Conv2d filter has a size of 1×1. This structure reduces the number of features while ensuring efficient encoding. After this, the same Basic2d module is connected to perform subsequent encoding.

[0075] The decoder structure of the generator is composed of De-Basic3d and Basic3d modules in an alternating series, wherein the De-Basic3d module comprises Deconv3d-BN-RReLU, and the Basic3d module comprises Conv3d-BN-RReLU.

[0076] In the generator, the mapping of two-dimensional data to three-dimensional data is completed through a dimension increasing operation. Specifically, the two-dimensional data obtained after encoding is first passed through a max-pooling layer, and then the data dimension is reconstructed to three dimensions using the view function. In order to further improve the reconstruction accuracy, a shortcut connection composed of Basic2d-Expand-Basic3d is introduced after each level of encoding operation, so as to copy and pass the information of different levels along the third dimension to the decoder.

[0077] In addition, the 3D discriminator in the present patent is composed of three Conv3d-BN-RReLU structures and a single Conv3d in series, which has the ability to distinguish true and false three-dimensional structures.

[0078] S4, based on the network model in S2, a citrus three-dimensional reconstruction model is trained: input the citrus X-ray picture in the data set, and map it to a three-dimensional CT structure through the 3D generator. Then, the 3D discriminator discriminates the generated three-dimensional data and returns the result to the generator. After receiving the feedback, the generator optimizes itself and then generates a three-dimensional CT structure with higher precision and transmits it to the discriminator again. Under the stimulation of more deceptive three-dimensional CT structures, the ability of the discriminator to distinguish true and false three-dimensional structures of citrus gradually improves. Finally, after alternating optimization training, the 3D generator and the 3D discriminator achieve Nash equilibrium, and the model can generate high-precision three-dimensional CT structures of citrus.

[0079] The citrus X-ray pictures in the data set are input into the 3D generator one by one, and the corresponding three-dimensional structures of citrus are generated after the encoding and decoding operations described in S3. Then, the discriminator discriminates the probability that the generated three-dimensional structure is true and returns the result to the generator. The generator optimizes itself according to the received feedback to generate a three-dimensional structure of citrus with higher precision. Then, the discriminator discriminates the more deceptive three-dimensional structure of citrus generated by the generator, and optimizes itself according to the discrimination accuracy. Finally, after alternating optimization training, the 3D generator and the 3D discriminator achieve Nash equilibrium, and the model can generate high-precision three-dimensional CT structures of citrus.

[0080] S5, input the citrus X-ray picture to be reconstructed into the trained citrus three-dimensional reconstruction model, and output the reconstructed 3D structure.

[0081] The to-be-tested citrus is placed on the production line conveyor belt one by one, and is driven into the X-ray machine. When the citrus moves to the detection area in the X-ray machine, the X-ray source emits X-rays, which pass through the to-be-tested citrus on the conveyor belt and fall on the detector. The detector converts the received signal into an electrical signal, which is further operated and imaged to obtain an X-ray picture of the citrus. The emission time interval of the X-rays is specified to ensure that an X-ray picture of each to-be-tested citrus is obtained. Finally, the citrus X-ray pictures are input into the three-dimensional reconstruction model obtained by training, and the corresponding three-dimensional structure of the citrus is output.

[0082] The technical solution can use only one X-ray image to complete three-dimensional reconstruction of fruits based on a self-built GAN three-dimensional reconstruction model, greatly simplifies the reconstruction process, shortens the reconstruction time, and can be applied to various real-time three-dimensional reconstruction scenes. It has the advantages of simple data acquisition process, fast reconstruction speed, etc.

[0083] Embodiment 2

[0084] Compared with embodiment 1, the embodiment has the following differences:

[0085] In S3 of embodiment 2, the 3D generator, in the encoder region, first inputs a pair of orthogonal images into a three-dimensional data space, saves the two-dimensional data orthogonality in the space, and sets the voxel values of the remaining positions to 0 to construct an orthogonal three-dimensional model of the citrus, as shown in Figure Two The input orthogonal three-dimensional model of the citrus is encoded by three Block3ds in the generator, and four RReLU-BN-Conv3ds are connected in series in each Block3d, wherein the second Conv3d has a filter size of 1x1x1. This structure can reduce the number of features while effectively encoding;

[0086] The decoder structure of the generator is composed of De-Basic3d and Basic3d connected in alternation, wherein the De-Basic3d module comprises Deconv3d-BN-RReLU, and the Basic3d module comprises Conv3d-BN-RReLU.

[0087] The input is an orthogonal three-dimensional model of the citrus, and after encoding and decoding processing, the corresponding three-dimensional model of the citrus is output. A fast connection composed of Basic3d is introduced after each level of encoding operation to pass different levels of information to the decoder. When the number of orthogonal image groups is greater than 1, the three-dimensional model of the citrus for each group of orthogonal images is output, and the weight of each three-dimensional model is calculated according to the total loss, as follows:

[0088]

[0089] The final three-dimensional model of the citrus is obtained by weighted average processing of the voxel values of each three-dimensional model of the citrus, as follows:

[0090]

[0091] wherein G * is the overall loss of the citrus fruit reconstructed three-dimensional model generator end, and sigma is a voxel value of the citrus fruit reconstructed three-dimensional model;

[0092] The 3D discriminator is composed of three Conv3d-BN-RReLU structures and a single Conv3d in series, and has the ability to distinguish true and false three-dimensional structures.

[0093] The embodiment reconstructs through multiple images, and has higher reconstruction accuracy.

[0094] The image acquisition stage includes a conveyor belt, an acquisition chamber and a fruit holder, the acquisition chamber is arranged on the conveyor belt, the acquisition chamber is respectively provided with an inlet and an outlet corresponding to the conveyor belt, the acquisition chamber is provided with a CCD camera, a radiation source and a detector, the conveyor belt is spaced apart and provided with a plurality of fruit holders, the fruit holder is rotationally matched with the conveyor belt, and the fruit holder is driven by a motor.

[0095] The patent can use only one X-ray image to complete fruit three-dimensional reconstruction based on the self-built GAN three-dimensional reconstruction model, greatly simplifies the reconstruction process, shortens the reconstruction time, and also can use multiple images for citrus three-dimensional reconstruction, although the time is longer, but the model accuracy of the reconstruction is higher.

[0096] The above only describes the preferred embodiments of the present application and is not intended to limit the present application, and any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for GAN-based three-dimensional reconstruction of X-ray images of citrus, characterized in that, It comprises the following steps: S1, obtaining a citrus X-ray picture, constructing a picture two-dimensional structure data set, and using CT reconstruction to reconstruct the corresponding three-dimensional structure of the citrus; the construction of the data set: using the X-ray projection pictures of the citrus at different angles to obtain a two-dimensional picture data set; using the CT reconstruction algorithm to reconstruct the three-dimensional structure of each citrus, and constructing a three-dimensional structure data set containing the corresponding three-dimensional structure of the citrus X-ray picture; S2, preprocessing the two-dimensional and three-dimensional data respectively; the preprocessing includes cropping and scaling of the X-ray film, format conversion, down-sampling, window width and window level adjustment of the CT data, so that the citrus X-ray picture and the corresponding three-dimensional structure of the citrus have the same perspective projection; the cropped picture is scaled to make the size of all the cropped citrus X-ray pictures consistent; S3, constructing a 3D reconstruction GAN network model: constructing a 3D generator and a 3D discriminator respectively; The 3DGAN model comprises: The game process is expressed by the following formula: where z is sampled from a noise distribution; since we want to learn the nonlinear mapping of x-rays to CT, the generated CT volume should be consistent with the semantic information provided by the input x-rays; The GAN discriminator loss is defined as: The generator loss is defined as: where x refers to the X-ray plane of the orange, and y refers to the corresponding CT volume. The reconstruction loss is defined as: The projection loss is: Given the definitions of the adversarial loss, the reconstruction loss, and the projection loss, our final objective function is expressed as: where λ1, λ2, and λ3control the relative importance of different loss terms; we set λ1= 0.1, λ2= λ3= 10; S4, training the citrus three-dimensional reconstruction model based on the network model in step 2; S5, inputting the to-be-reconstructed citrus X-ray picture into the trained citrus three-dimensional reconstruction model to output the reconstructed 3D structure.

2. The three-dimensional reconstruction method of the citrus X-ray image based on GAN according to claim 1, wherein S3, in the encoder region, the input citrus X-ray picture first passes through a Basic2d module composed of Conv2d-BN-RReLU in series in the generator; then three Blocks are executed for coding, and four RReLU-BN-Conv2d are connected in series in each Block, wherein the second Conv2d filter size is 1x1, which can reduce the feature quantity on the premise of effective coding, and an identical Basic2d module is connected to realize subsequent coding.

3. The three-dimensional reconstruction method of the citrus X-ray image based on GAN according to claim 1, wherein the decoder structure of the generator is composed of De-Basic3d and Basic3d connected in series alternately, wherein the De-Basic3d module comprises Deconv3d-BN-RReLU, and the Basic3d module comprises Conv3d-BN-RreLU; In the generator, the mapping of two-dimensional data to three-dimensional data is completed through a dimension increasing operation; specifically, the two-dimensional data obtained after coding first passes through a maximum pooling layer, then the data dimension is reconstructed to three-dimensional using the view function, and a shortcut connection composed of Basic2d-Expand-Basic3d is introduced after each level of coding operation to copy the information at different levels along the third dimension and pass it to the decoder.

4. The three-dimensional reconstruction method of the citrus X-ray image based on GAN according to claim 1, wherein S4, the citrus X-ray pictures in the data set are input into the 3D generator one by one, and the corresponding three-dimensional structure of the citrus is generated after the coding and decoding operation of S3; Then, the discriminator discriminates the probability that the generated three-dimensional structure is true and returns the result to the generator; The generator performs self-optimization according to the received feedback to generate a three-dimensional structure of the citrus with higher precision. ​ ​ ​ Then, the discriminator discriminates the newly generated more deceptive citrus three-dimensional structure, and is self-optimized according to the discrimination accuracy; finally, after alternating optimization training, the 3D generator and the 3D discriminator realize Nash equilibrium, and the model can generate high-precision citrus three-dimensional CT structure.

5. The GAN-based three-dimensional reconstruction method for citrus X-ray images according to claim 1, characterized in that: wherein S3, the 3D generator, in the encoder region, firstly inputs a pair of orthogonal images into a three-dimensional data space, saves the two-dimensional data orthogonality in the space, sets the remaining position voxel values to 0, and constructs a citrus orthogonal three-dimensional model; the input citrus orthogonal three-dimensional model is encoded by three Block3ds in the generator, and four RReLU-BN-Conv3ds are connected in series in each Block3d, wherein the second Conv3d filter size is 1x1x1, and the structure can reduce the feature quantity on the premise of effective coding; The decoder structure of the generator is alternatively connected by De-Basic3d and Basic3d, wherein the De-Basic3d module comprises Deconv3d-BN-RReLU, and the Basic3d module comprises Conv3d-BN-RReLU. The input is an orthogonal three-dimensional model of a citrus fruit, and after encoding and decoding processing, the corresponding reconstructed three-dimensional model of the citrus fruit is output. A fast connection composed of Basic3d is introduced after each level of encoding operation to pass information of different levels to the decoder. When the number of orthogonal image groups is greater than 1, the reconstructed three-dimensional model of the citrus fruit of each group of orthogonal images is output, and the weight of each three-dimensional model is calculated according to the total loss, as follows: The final three-dimensional model of the citrus fruit is obtained by weighted average processing of the voxel values of each reconstructed three-dimensional model of the citrus fruit, as follows: wherein, is the total loss of the citrus fruit reconstructed three-dimensional model generator end, is the voxel value of the citrus fruit reconstructed three-dimensional model. The 3D discriminator is composed of three Conv3d-BN-RReLU structures and a single Conv3d.

6. A carrier platform for implementing the GAN-based three-dimensional reconstruction method for citrus X-ray images according to claim 1, characterized in that: It comprises a conveyor belt, an acquisition chamber and a fruit holder, the acquisition chamber is arranged on the conveyor belt, the acquisition chamber is provided with an inlet and an outlet corresponding to the conveyor belt, the acquisition chamber is provided with a CCD camera, a radiation source and a detector, the conveyor belt is provided with a plurality of fruit holders at intervals, the fruit holder rotates relative to the conveyor belt, and the fruit holder is driven by a motor.

Images