Microscopic three-dimensional shape reconstruction method of high-speed deep network
Through the high-speed deep network method, the packet convolution and jump connection layers are used to process different focal surface images of microscopic scenes, which solves the problem of long time for microscopic scene reconstruction in the existing technology, and achieves efficient three-dimensional morphological reconstruction and accuracy improvement.
Patent Information
- Application Number
- CN202411240088.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-05
- Publication Date
- 2024-12-27
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing three-dimensional morphological reconstruction technology cannot meet the high-resolution requirements of microscopic scenarios, resulting in a significant increase in model inference time and the real-time requirements cannot be achieved.
Using high-speed deep network method, by acquiring images of different focal surfaces of microscopic scenes, using grouping convolution and jump connection layers, combining encoder and decoder for feature extraction and fusion, filtering and reconstruction of depth maps, reducing the parameters of neural networks and calculation quantity to realize real-time reconstruction.
It effectively reduces the inference time of neural networks and improves the reconstruction accuracy and real-timeness of microscopic three-dimensional morphology.
Smart Images
Figure CN119206056A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a high-speed deep network microscopic three-dimensional morphology reconstruction method. Background Art
[0002] Microscopic 3D shape reconstruction is mainly used in defect modeling and detection of precision products. Compared with 2D scene graphs, 3D shape graphs can restore the structural features in microscopic scenes more comprehensively and realistically. However, existing 3D shape reconstruction mainly solves the depth recovery of macroscopic scenes, and the data collected from macroscopic scenes cannot be applied to the high-resolution characteristics of microscopic scenes. In addition, as the resolution of the input image sequence increases, the model inference time will increase significantly, which cannot meet the real-time requirements in microscopic scenes.
[0003] Based on the above technical problems, this application proposes a microscopic three-dimensional morphology reconstruction method based on a high-speed deep network. Summary of the invention
[0004] The purpose of the present invention is to provide a high-speed deep network microscopic three-dimensional morphology reconstruction method to solve the technical problems mentioned in the background technology. The purpose of the present invention is achieved through the following technical solutions:
[0005] A high-speed deep network microscopic three-dimensional shape reconstruction method comprises the following steps:
[0006] Step S1, obtaining N images of a microscopic scene with the same interval L and different focal planes;
[0007] Step S2, segmenting the N images according to a size of H×W in height and width to generate a tensor with N channels and H×W in height and width;
[0008] Step S3, inputting the tensor obtained in step S2 into the encoder to obtain an extracted feature tensor;
[0009] Step S4, inputting the extracted feature tensor into a decoder to obtain a fused feature tensor;
[0010] Step S5, multiplying the fused feature tensor obtained in step S4 by the interval L in step S1, and obtaining a depth map after filtering;
[0011] Step S6: reassemble the depth map obtained in step S5 according to the segmentation order.
[0012] Furthermore, the encoder is divided into four levels from top to bottom. The first three levels each include two first convolutional layers, a ReLU activation function, and a downsampling layer; the fourth level includes two second convolutional layers and a ReLU activation function.
[0013] Furthermore, the convolution kernel size of the first convolution layer is 7, the padding variable is 3, and the convolution step is 1; the number of convolution kernels of the two first convolution layers of the first three levels are 16 and 32, 32 and 64, and 64 and 128 respectively.
[0014] Furthermore, the convolution kernel size of the second convolution layer is 3, the padding variable is 1, and the convolution step is 1; the number of convolution kernels of the two second convolution layers are 128 and 128 respectively.
[0015] Furthermore, the downsampling layer is a maximum pooling layer, and the step size of the maximum pooling layer is 2×2.
[0016] Furthermore, the decoder is divided into three layers from bottom to top. Each layer includes an upsampling layer, a skip connection layer, two third convolutional layers and a ReLU activation function. The skip connection layer adds the information of the same size in the encoder to the corresponding decoded information.
[0017] Furthermore, the upsampling layer adopts a nearest neighbor difference algorithm, and the step size of the upsampling layer is 2×2.
[0018] Furthermore, the convolution kernel size of the third convolution layer is 7, the padding variable is 3, and the convolution step is 1; the number of convolution kernels of the two first convolution layers of the three levels are 128 and 64, 64 and 32, and 8 and 1 respectively.
[0019] Furthermore, the filtering in step S5 is statistical filtering.
[0020] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:
[0021] 1. Through group convolution, the number of parameters and computation of the neural network model is effectively reduced, the reasoning time of the Shenjiang network is reduced, and the real-time reconstruction of microscopic three-dimensional morphology is achieved;
[0022] 2. The information of the same size in the encoding stage is added to the corresponding decoding information through the jump connection layer, which avoids the gradient disappearance in the neural network training and improves the reconstruction accuracy of the microscopic scene. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] The drawings in the specification, which constitute a part of this application, are used to provide a further understanding of the present invention. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.
[0024] Figure 1 This is a flow chart of an embodiment of the present application. DETAILED DESCRIPTION
[0025] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the drawings and specific implementation methods of the specification. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0026] like Figure 1 A high-speed deep network microscopic three-dimensional morphology reconstruction method is shown, comprising the following steps:
[0027] Step S1, adjust the vertical distance from the camera to the microscopic scene, change the vertical distance from the microscopic scene to the camera's photosensitive imaging surface, and obtain N images of the microscopic scene at different focal planes at the same interval L. In order to ensure the signal-to-noise ratio of the collected data, this method collects grayscale images with an image size of 5120×4096 or 4096×3072 or other.
[0028] Step S2: segment the obtained image according to a size of 512×512 in height and width to obtain image units with a size of 512×512 in height and width, and merge the obtained image units into a tensor with N channels and a size of 512×512 in height and width.
[0029] Step S3: input the tensor obtained in step S2 into the encoder to obtain the extracted feature tensor.
[0030] The encoder is divided into four layers from top to bottom. The first three layers include two first convolutional layers, ReLU activation function and downsampling layer. The convolution kernel size of the first convolutional layer is 7, the padding variable is 3, and the convolution step is 1. The number of convolution kernels of the two first convolutional layers of the first three layers is 16 and 32, 32 and 64, 64 and 128 respectively. The downsampling layer is a maximum pooling layer, and the step size of the maximum pooling layer is 2×2.
[0031] The fourth level includes two second convolutional layers and ReLU activation function. The convolution kernel size of the second convolutional layer is 3, the padding variable is 1, and the convolution step is 1. The number of convolution kernels of the two second convolutional layers is 128 and 128 respectively.
[0032] Step S3 specifically includes:
[0033] Step S31, perform two-step convolution and ReLU activation function operations on the tensor obtained in step S2. The size of the convolution kernel used is 7, the padding variable used is 3, the convolution step is 1, the convolution kernels of the two-step convolution are 16 and 32 respectively, and a tensor with H=512, W=512 and 32 channels is obtained.
[0034] Step S32, using a maximum pooling layer with a step size of 2x2 to downsample the tensor obtained in step S31 to obtain a tensor with H=256, W=256 and 32 channels;
[0035] Step S33, perform two-step convolution and ReLU activation function operations on the tensor obtained in step S32. The size of the convolution kernel used is 7, the padding variable used is 3, the convolution step is 1, the convolution kernels of the two-step convolution are 32 and 64 respectively, and a tensor with H=256, W=256 and 64 channels is obtained.
[0036] Step S34, using a maximum pooling layer with a step size of 2x2 to downsample the tensor obtained in step S33 to obtain a tensor with H=128, W=128 and 64 channels;
[0037] Step S35: Perform two-step convolution and ReLU activation function operations on the tensor obtained in step S34. The size of the convolution kernel used is 7, the padding variable used is 3, the convolution step is 1, and the convolution kernels of the two-step convolution are 64 and 128 respectively, obtaining a tensor with H=128, W=128 and 128 channels.
[0038] Step S36, using a maximum pooling layer with a step size of 2x2 to downsample the tensor obtained in step S35 to obtain a tensor with H=64, W=64 and 128 channels;
[0039] Step S37, perform two-step convolution and ReLU activation function operations on the tensor obtained in step S36. The size of the convolution kernel used is 3, the padding variable used is 1, the convolution step is 1, and the convolution kernels of the two-step convolution are 64 and 64 respectively, obtaining a tensor with H=64, W=64 and 128 channels.
[0040] Step S4, inputting the extracted feature tensor into a decoder to obtain a fused feature tensor;
[0041] The decoder is divided into three layers from bottom to top. Each layer includes an upsampling layer, a skip connection layer, two third convolution layers and a ReLU activation function. The skip connection layer adds the information of the same size in the encoder to the corresponding decoded information. The upsampling layer uses the nearest neighbor difference algorithm, and the step size of the upsampling layer is 2×2. The convolution kernel size of the third convolution layer is 7, the padding variable is 3, and the convolution step size is 1; the number of convolution kernels of the two first convolution layers of the three levels are 128 and 64, 64 and 32, and 8 and 1 respectively.
[0042] Step S4 specifically includes:
[0043] Step S41, use the nearest neighbor difference algorithm function with a step size of 2x2 to upsample the tensor obtained in step S37 to obtain a tensor with H=128, W=128 and 128 channels;
[0044] Step S42, merging the tensor obtained in step S41 with the tensor obtained in step S35 for the third channel to obtain a tensor with H=128, W=128 and 256 channels;
[0045] In step S43, the tensor obtained in step S42 is subjected to two-step convolution and ReLU activation function operations, the size of the convolution kernel is 7, the padding variable is 3, the convolution step is 1, the convolution kernels of the two-step convolution are 128 and 64 respectively, and a tensor with H=128 and W=128 and 64 channels is obtained.
[0046] Step S44, up-sample the tensor obtained in step S43 using a nearest neighbor difference algorithm function with a step size of 2x2 to obtain a tensor with H=256, W=256 and 64 channels;
[0047] Step S45, merging the tensor obtained in step S44 with the tensor obtained in step S33 for the third channel to obtain a tensor with H=256, W=256 and 128 channels;
[0048] Step S46, perform two-step convolution and ReLU activation function operations on the tensor obtained in step S45, use a convolution kernel size of 7, a padding variable of 3, a convolution step of 1, and two-step convolution kernels of 64 and 32 respectively, to obtain a tensor with H=256 and W=256 and a channel number of 32.
[0049] Step S47, use the nearest neighbor difference algorithm function with a step size of 2x2 to upsample the tensor obtained in step S46 to obtain a tensor with H=512, W=512 and 32 channels;
[0050] Step S48, merging the tensor obtained in step S47 with the tensor obtained in step S31 for the third channel to obtain a tensor with H=512, W=512 and 64 channels;
[0051] Step S49, perform two-step convolution and ReLU activation function operations on the tensor obtained in step S48, use a convolution kernel size of 7, a padding variable of 3, a convolution step of 1, and two-step convolution kernels of 8 and 1 respectively, to obtain a tensor with H=512 and W=512 and a channel number of 1.
[0052] Step S5: multiply the fused feature tensor obtained in step S4 by the interval L in step S1, and obtain a depth map after performing a statistical filtering.
[0053] The statistical filtering method smoothes the data and removes noise by calculating the statistical information (such as mean, standard deviation, etc.) in the neighborhood around each point in the point cloud. Specifically, it calculates a statistical metric for each point (in this example, the average distance to the 100 nearby points), and then compares this metric with a preset threshold. If the statistical metric of a point exceeds the threshold, the point is considered a noise point or an outlier and is removed from the point cloud.
[0054] Step S6: reassemble the depth map obtained in step S5 according to the segmentation order to obtain a three-dimensional depth map of the microscopic scene.
[0055] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:
[0056] 1. Through group convolution, the number of parameters and computation of the neural network model is effectively reduced, the reasoning time of the Shenjiang network is reduced, and the real-time reconstruction of microscopic three-dimensional morphology is achieved;
[0057] 2. The information of the same size in the encoding stage is added to the corresponding decoding information through the jump connection layer, which avoids the gradient disappearance in the neural network training and improves the reconstruction accuracy of the microscopic scene.
[0058] It should be noted that the use of terms such as "first" and "second" to limit components is only to facilitate the distinction between corresponding components. Unless otherwise stated, the above terms have no special meaning and therefore cannot be understood as limiting the scope of protection of the present invention.
[0059] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A high-speed deep network microscopic three-dimensional shape reconstruction method, characterized in that: The following steps are involved: Step S1, obtaining N images of a microscopic scene with the same interval L and different focal planes; Step S2, segmenting the N images according to a size of H×W in height and width to generate a tensor with N channels and H×W in height and width; Step S3, inputting the tensor obtained in step S2 into the encoder to obtain an extracted feature tensor; Step S4, inputting the extracted feature tensor into a decoder to obtain a fused feature tensor; Step S5, multiplying the fused feature tensor obtained in step S4 by the interval L in step S1, and obtaining a depth map after filtering; Step S6: reassemble the depth map obtained in step S5 according to the segmentation order.
2. According to the high-speed deep network microscopic three-dimensional shape reconstruction method of claim 1, it is characterized in that: The encoder is divided into four levels from top to bottom. The first three levels each include two first convolutional layers, a ReLU activation function and a downsampling layer; the fourth level includes two second convolutional layers and a ReLU activation function.
3. According to the high-speed deep network microscopic three-dimensional shape reconstruction method of claim 2, it is characterized in that: The convolution kernel size of the first convolution layer is 7, the padding variable is 3, and the convolution step is 1; the number of convolution kernels of the two first convolution layers of the first three levels are 16 and 32, 32 and 64, and 64 and 128 respectively.
4. According to the high-speed deep network microscopic three-dimensional shape reconstruction method of claim 2, it is characterized in that: The convolution kernel size of the second convolution layer is 3, the padding variable is 1, and the convolution step size is 1; the number of convolution kernels of the two second convolution layers are 128 and 128 respectively.
5. According to the high-speed deep network microscopic three-dimensional shape reconstruction method of claim 2, it is characterized in that: The downsampling layer is a maximum pooling layer, and the step size of the maximum pooling layer is 2×2.
6. The high-speed deep network microscopic three-dimensional shape reconstruction method according to claim 1, characterized in that: The decoder is divided into three layers from bottom to top, each layer includes an upsampling layer, a skip connection layer, two third convolutional layers and a ReLU activation function, and the skip connection layer adds the information of the same size in the encoder to the corresponding decoding information.
7. The high-speed deep network microscopic three-dimensional shape reconstruction method according to claim 6 is characterized in that: The upsampling layer adopts a nearest neighbor difference algorithm, and the step size of the upsampling layer is 2×2.
8. The high-speed deep network microscopic three-dimensional shape reconstruction method according to claim 6, characterized in that: The convolution kernel size of the third convolution layer is 7, the padding variable is 3, and the convolution step size is 1; the numbers of convolution kernels of the two first convolution layers of the three levels are 128 and 64, 64 and 32, and 8 and 1 respectively.
9. The high-speed deep network microscopic three-dimensional shape reconstruction method according to claim 1, characterized in that: The filtering in step S5 is statistical filtering.