Quasi-three-dimensional conditional adversarial network training method and application method thereof in three-dimensional reconstruction

Abstract

The invention relates to a method for training a quasi-three-dimensional conditional adversarial network and applying the quasi-three-dimensional conditional adversarial network to three-dimensional reconstruction, and belongs to the field of three-dimensional reconstruction. By means of the quasi-three-dimensional processing method of the three-dimensional structure, the dimension of the data is reduced, meanwhile, the feature information of the three-dimensional data is included, the training efficiency of the data is improved, and computing resources needed by data training are reduced. Meanwhile, in combination with a three-dimensional structure simulation method, the quasi-three-dimensional conditional adversarial network model is applied to three-dimensional reconstruction, and the precision and accuracy of the three-dimensional structure are improved.

Description

Technical Field

[0001] The invention belongs to the field of three-dimensional reconstruction, and in particular relates to a method for training a quasi-three-dimensional conditional adversarial network and applying it in three-dimensional reconstruction. Background Art

[0002] Conditional adversarial network is a deep generative model, which is widely used in image generation, video generation and text generation. It is a relatively cutting-edge deep learning method. Conditional adversarial network is usually trained with two-dimensional images as training sets. In the literature IEEE Int.Conf.Comput.Vis., 2242–2251 (2017). and the literature IEEEConf.Comput.Vis.Pattern Recognit., 5967–5976 (2017)., researchers have achieved good image generation results. In the patents "Super-resolution generation and conditional adversarial network remote sensing image sample generation method" and "A super-resolution reconstruction method based on conditional generative adversarial network", researchers also used this method to complete the generation and optimization of two-dimensional images. However, due to the increase in data dimensions, the difficulty and resources of training will increase significantly in three-dimensional training and application, so there is a lack of research in this area. Due to the lack of corresponding research on three-dimensional conditional adversarial networks, there is also a lack of related applications.

[0003] Three-dimensional reconstruction is an important characterization technology in the fields of materials science, biology, medicine, etc. The construction of three-dimensional data and the optimization of three-dimensional models will greatly affect the accuracy of researchers' understanding of material structure. The patent "A deep learning method for reconstructing transmission electron tomography images of micro-nano structures" uses deep learning technology to optimize the structure of electron tomography image reconstruction. Although it solves the problem of image generation quality, its training method is also two-dimensional images, and there is a lack of research and application of three-dimensional training methods. Summary of the invention

[0004] The purpose of the present invention is to overcome the difficulty of three-dimensional training of conditional adversarial networks, and to provide a quasi-three-dimensional conditional adversarial network training and its application method in three-dimensional reconstruction, which greatly improves the accuracy and efficiency of three-dimensional structure generation and can be applied to accurate three-dimensional structure reconstruction.

[0005] To achieve the above object, the technical solution of the present invention is: a method for training a quasi-three-dimensional conditional adversarial network and its application in three-dimensional reconstruction, comprising:

[0006] Generation of three-dimensional structures;

[0007] Quasi-three-dimensional data processing;

[0008] Data training;

[0009] Applications of 3D reconstruction.

[0010] In one embodiment of the present invention, the three-dimensional structure is generated by using a spatial pixel expansion algorithm.

[0011] In one embodiment of the present invention, the generation of the three-dimensional structure is specifically implemented as follows: first, a cubic three-dimensional matrix is ​​generated; to ensure accurate processing of boundary data, a predetermined blank value is reserved in each direction, and a predetermined number of pixel points are randomly generated in the matrix, and the value of each pixel is 1 to 255, indicating different diffusion capabilities; a three-dimensional Gaussian filter function is used to simulate the growth process of the structure; to distinguish the contour of the three-dimensional structure, a spherical space restriction is used; pixels in the matrix that are less than a certain value are set to 0, and a cleaning process is simulated to obtain a structure matrix No. 1; then, an internal etching structure is simulated to generate a spherical inner cavity, a predetermined number of random points are generated in the spherical inner cavity area, and a three-dimensional Gaussian filter function is used to grow the data, and pixels in the matrix that are less than a certain value are set to 0 to obtain a structure matrix No. 2; the sum of matrix No. 1 and matrix No. 2 is calculated, which is the three-dimensional structure.

[0012] In one embodiment of the present invention, the pixel value of each point in the three-dimensional structure is expressed as:

[0013] G(x,y,z)=max(a0 / (2πσ^2)·e^((-[(xx c )^2+(yy c )^2+(zz c )^2]) / 2σ2)-ε,0)

[0014] Among them, (x, y, z) are the coordinates of the pixel point, (x c ,y c ,z c ) is the center position coordinate of the Gaussian kernel, a0 is the set pixel value, σ is the standard deviation of the Gaussian kernel, and ε is the small value to be filtered.

[0015] In one embodiment of the present invention, the quasi-three-dimensional data processing adopts a two-dimensional processing algorithm for three-dimensional data under normalized constraints.

[0016] In one embodiment of the present invention, the quasi-three-dimensional data processing is specifically implemented as follows: first, the three-dimensional structure is two-dimensionalized according to slice data or projection data to obtain a two-dimensional data set containing three-dimensional feature information; then, each two-dimensional data set is normalized as an independent unit to obtain a data set with independent normalized features.

[0017] In one embodiment of the present invention, the data is trained using a conditional adversarial network model for image-to-image conversion.

[0018] In one embodiment of the present invention, the specific implementation method of the data training is: first, training feature processing is performed on a data set of independent normalized features, and the feature-processed data set and a two-dimensional data set of three-dimensional feature information form a training data pair to perform conditional adversarial network model training.

[0019] In one embodiment of the present invention, the application of the three-dimensional reconstruction is to use the conditional adversarial network model trained in data training to perform data optimization.

[0020] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0021] Compared with the prior art, the present invention has the following beneficial effects: the method of the present invention uses a quasi-three-dimensional processing method for three-dimensional structures to reduce the dimension of data, while including the characteristic information of three-dimensional data, improving the training efficiency of data and reducing the computing resources required for data training. At the same time, combined with the three-dimensional structure simulation method, the quasi-three-dimensional conditional adversarial network model is applied to three-dimensional reconstruction, improving the precision and accuracy of the three-dimensional structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 For one hundred three-dimensional simulation structures.

[0023] Figure 2 The three-dimensional simulation structure and its two-dimensional feature processing results; a, the original simulation structure; b, three projection data with added noise; c, all projection data.

[0024] Figure 3 For three-dimensional reconstruction applications; where a is the original simulation structure; b is the three-dimensional structure reconstructed by three tilt sequences; c is the three-dimensional structure obtained by the quasi-3D conditional adversarial network model I; d is the high-resolution three-dimensional structure obtained by the quasi-3D conditional adversarial network model II. DETAILED DESCRIPTION

[0025] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0026] like Figure 3 As shown, the present invention provides a method for training a quasi-three-dimensional conditional adversarial network and applying it in three-dimensional reconstruction, including:

[0027] Generation of three-dimensional structures;

[0028] Quasi-three-dimensional data processing;

[0029] Data training;

[0030] Applications of 3D reconstruction.

[0031] The generation of the three-dimensional structure adopts a spatial pixel expansion algorithm. The specific implementation method is as follows: first, generate a cubic three-dimensional matrix; to ensure accurate processing of boundary data, retain a predetermined blank value in each direction, randomly generate a predetermined number of pixel points in the matrix, and the value of each pixel is 1 to 255, indicating different diffusion capabilities; use a three-dimensional Gaussian filter function to simulate the growth process of the structure; to distinguish the contour of the three-dimensional structure, use a spherical space restriction; set the pixels in the matrix that are less than a certain value to 0, simulate the cleaning process, and obtain the No. 1 structure matrix; then, simulate the internal etching structure, generate a spherical inner cavity, generate a predetermined number of random points in the spherical inner cavity area, and use a three-dimensional Gaussian filter function to grow the data, set the pixels in the matrix that are less than a certain value to 0, and obtain the No. 2 structure matrix; calculate the sum of the No. 1 matrix and the No. 2 matrix, which is the three-dimensional structure.

[0032] The pixel value of each point in the three-dimensional structure is expressed as:

[0033] G(x,y,z)=max(a0 / (2πσ^2)·e^((-[(xx c )^2+(yy c )^2+(zz c )^2]) / 2σ2)-ε,0)

[0034] Among them, (x, y, z) are the coordinates of the pixel point, (x c ,y c ,z c ) is the center position coordinate of the Gaussian kernel, a0 is the set pixel value, σ is the standard deviation of the Gaussian kernel, and ε is the small value to be filtered.

[0035] The quasi-3D data processing adopts a two-dimensional processing algorithm for three-dimensional data under normalized constraints. The specific implementation method is: first, the three-dimensional structure is processed in two dimensions according to the slice data or projection data to obtain a two-dimensional data set containing three-dimensional feature information; then, each two-dimensional data set is normalized as an independent unit to obtain a data set with independent normalized features.

[0036] The training of the data adopts a conditional adversarial network model of image-to-image conversion. The specific implementation method is: first, the training feature processing is performed on the data set of independent normalized features, and the feature-processed data set and the two-dimensional data set of three-dimensional feature information form a training data pair, and the conditional adversarial network model training is performed.

[0037] The application of the three-dimensional reconstruction is to use the conditional adversarial network model trained in data training to perform data optimization.

[0038] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0039] The training and application of quasi-3D conditional adversarial networks can expand the application field of deep learning models. The quasi-3D processing and training method proposed in the present invention can improve the training speed of the model and avoid the dimensionality explosion of the data. This processing method can also be expanded to higher dimensions, which can greatly reduce the occupation of computing resources. For a low-resolution three-dimensional structure of 256×256×256, quasi-3D data training only needs to meet 256×256 computing resources to perform model training, while three-dimensional training needs to be increased by 256 times, which can meet many lightweight training needs.

[0040] In the process of 3D reconstruction, we often encounter problems such as insufficient training data and overfitting or underfitting of deep training models. The use of this quasi-3D data training model can effectively avoid these problems. By quickly and efficiently simulating the 3D structure and performing characterization processing, the underfitting problem can be effectively avoided; the quasi-3D processing method can effectively avoid the overfitting problem by converting the 3D data into two dimensions under normalized constraints.

[0041] Example 1

[0042] Example of generating a three-dimensional structure:

[0043] The size of the three-dimensional matrix of the cube is 178×178×178. 20 blank pixels are reserved in each direction, and the actual vertices of the three-dimensional body are random values ​​between 20 and 158. A certain number of pixels (70 to 700) are randomly generated in the matrix, and the value of each pixel is 1 to 255. The three-dimensional Gaussian filter function is used to simulate the diffusion process of particles. Some structures are restricted by a spherical space with a radius of 50 to 70 pixels. Pixels in the matrix that are less than a certain value (a random value in [0.05,0.1]) are set to 0, and the resulting structure is called structure No. 1.

[0044] Simulate the internal etching structure and generate a spherical cavity with a spherical radius of 20 to 35 pixels. Generate 15 to 150 random points in this area and grow the data using a 3D Gaussian filter function. Set the pixels in the matrix that are less than a certain value (a random value in [0.05, 0.1]) to 0, and the resulting structure is called structure 2. Calculate the sum of the matrices of structures 1 and 2.

[0045] Finally, the value of the spatial point is:

[0046] G(x,y,z)=max(a0 / (2πσ^2)·e^((-[(xxc )^2+(yy c )^2+(zz c )^2]) / 2σ2)-ε,0)

[0047] Where (x, y, z) is the coordinate of the pixel point. c ,y c ,z c ) are the center position coordinates of the Gaussian kernel (a0∈[1,255]), σ∈[3,10] are the standard deviation of the Gaussian kernel, and ε∈[0.05,0.1] are the small values ​​to be filtered.

[0048] Repeat the above process to simulate one hundred three-dimensional structures, such as Figure 1 shown.

[0049] Example 2

[0050] Projection data processing of three-dimensional structures and training of quasi-three-dimensional conditional adversarial networks, as well as application examples of three-dimensional reconstruction using trained models.

[0051] Using Radon transform, each slice of the three-dimensional structure is transformed into a projection sinusoidal graph A fig The transformation angle is set from -90° to 90°, the step size is 0.7°, and each 3D structure is normalized separately. Extract the sinusoidal graphs B at -70°, 0°, and 70° fig ,In order to simulate the real structural test results, Gaussian noise, Poisson noise and image blur (such as Figure 2 b), the three images are used for 3D reconstruction to obtain the pre-built model. The pre-built 3D structure is again subjected to Radon transformation to obtain the projection sinusoidal graph C fig , and each 3D structure was normalized separately. These data were paired with the Radon transformed images of the original structure, and the pix2pix network was trained on the paired data for two consecutive 400 cycles.

[0052] Apply the trained network to the sinusoidal graph C fig , or other image sets with missing data, to obtain the complete sinusoidal graph D fig , and perform three-dimensional reconstruction to obtain an accurate three-dimensional structure.

[0053] Example 3

[0054] Slice data processing of three-dimensional structures and training of quasi-three-dimensional conditional adversarial networks, as well as application examples of three-dimensional reconstruction using trained models.

[0055] The three-dimensional structure is divided into 256 slices, and the slice map E is obtained. fig, each 3D structure is normalized separately. Gaussian noise, Poisson noise and image blur are added to the slice image to obtain image F fig The processed slice data and the slices of the original structure form data pairs, and these pairs are trained for 500 cycles using the pix2pix network. The trained network is applied to the sinusoidal graph E fig , or other image sets with fuzzy data, to obtain an accurate slice map G fig , reconstruct all slices to obtain an accurate three-dimensional structure.

[0056] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, belong to the protection scope of the present invention.

Claims

1. A method for training a quasi-three-dimensional conditional adversarial network and its application in three-dimensional reconstruction, characterized in that: include: Generation of three-dimensional structures; Quasi-three-dimensional data processing; Data training; Applications of 3D reconstruction.

2. The method for training a quasi-three-dimensional conditional adversarial network and its application in three-dimensional reconstruction according to claim 1, characterized in that: The three-dimensional structure is generated by using a spatial pixel expansion algorithm.

3. The method for training a quasi-three-dimensional conditional adversarial network and its application in three-dimensional reconstruction according to claim 1 or 2, characterized in that: The specific implementation method of generating the three-dimensional structure is as follows: first, generate a cubic three-dimensional matrix; to ensure accurate processing of boundary data, retain a predetermined blank value in each direction, randomly generate a predetermined number of pixel points in the matrix, and the value of each pixel is 1 to 255, indicating different diffusion capabilities; use a three-dimensional Gaussian filter function to simulate the growth process of the structure; to distinguish the outline of the three-dimensional structure, use a spherical space restriction; set pixels in the matrix that are less than a certain value to 0, simulate the cleaning process, and obtain a structure matrix No. 1; then, simulate the internal etching structure, generate a spherical inner cavity, generate a predetermined number of random points in the spherical inner cavity area, and use a three-dimensional Gaussian filter function to grow the data, set pixels in the matrix that are less than a certain value to 0, and obtain a structure matrix No. 2; calculate the sum of matrix No. 1 and matrix No. 2, which is the three-dimensional structure.

4. The method for training a quasi-three-dimensional conditional adversarial network and its application in three-dimensional reconstruction according to claim 3, characterized in that: The pixel value of each point in the three-dimensional structure is expressed as: G(x,y,z)=max(a0 / (2πs^2);e^((-[(xx c )^2+(yy c )^2+(zz c )^2]) / 2σ2)-ε,0) Among them, (x, y, z) are the coordinates of the pixel point, (x c ,y c ,z c ) is the center position coordinate of the Gaussian kernel, a0 is the set pixel value, σ is the standard deviation of the Gaussian kernel, and ε is the small value to be filtered.

5. The method for training a quasi-three-dimensional conditional adversarial network and its application in three-dimensional reconstruction according to claim 1, characterized in that: The quasi-three-dimensional data processing adopts a two-dimensional processing algorithm for three-dimensional data under normalized constraints.

6. The method for training a quasi-three-dimensional conditional adversarial network and its application in three-dimensional reconstruction according to claim 1 or 5, characterized in that: The specific implementation method of the quasi-three-dimensional data processing is: first, the three-dimensional structure is two-dimensionalized according to the slice data or projection data to obtain a two-dimensional data set containing three-dimensional feature information; then, each two-dimensional data set is normalized as an independent unit to obtain a data set with independent normalized features.

7. The method for training a quasi-three-dimensional conditional adversarial network and its application in three-dimensional reconstruction according to claim 1, characterized in that: The data is trained using a conditional adversarial network model for image-to-image translation.

8. The method for training a quasi-three-dimensional conditional adversarial network and its application in three-dimensional reconstruction according to claim 6, characterized in that: The specific implementation method of the data training is: first, training feature processing is performed on a data set of independent normalized features, and the feature-processed data set and a two-dimensional data set of three-dimensional feature information form a training data pair to perform conditional adversarial network model training.

9. The method for training a quasi-three-dimensional conditional adversarial network and its application in three-dimensional reconstruction according to claim 1, characterized in that: The application of the three-dimensional reconstruction is to use the conditional adversarial network model trained in data training to perform data optimization.

10. A computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, and when the processor executes the computer program instructions, the method steps according to any one of claims 1 to 9 can be implemented.

Images