Multi-scale microscopic three-dimensional morphology reconstruction method with embedded depth-of-field relative position codes

Abstract

The invention belongs to the technical field of three-dimensional reconstruction, and particularly relates to a multi-scale microscopic three-dimensional shape reconstruction method with embedded depth-of-field relative position codes. Comprising the following steps: firstly, collecting a multi-depth-of-field image sequence in a microscopic scene through a microscopic camera, and taking a collected coordinate or sequence number as a depth-of-field position; secondly, a focusing feature extraction module is constructed to extract focusing information of the multi-depth-of-field image sequence, and a layer attention module is used for optimization; then, the depth-of-field position is coded and embedded into a focusing feature, and a focusing multi-scale feature is extracted; and finally, mapping into a three-dimensional morphology graph through feature convolution. The depth-of-field position coding and the multi-scale feature extraction provided by the invention can effectively combine the focusing relationship and the scale relationship between the depth-of-field image sequences so as to accurately express the three-dimensional shape of the scene.

Description

Technical Field

[0001] The invention belongs to the technical field of three-dimensional reconstruction, and in particular relates to a multi-scale microscopic three-dimensional morphology reconstruction method embedded with depth of field relative position coding. Background Art

[0002] Microscopic three-dimensional morphology reconstruction is a core research field of precision measurement. In recent years, with the continuous evolution of artificial intelligence and machine vision, cross-research methods have pushed microscopic three-dimensional morphology reconstruction methods to a new level and are widely used in cutting-edge scientific and technological fields such as precision parts processing and medical biological quantitative analysis.

[0003] In the era of intelligence and information technology, the iterative progress of industrial transformation has put forward new requirements for microscopic three-dimensional reconstruction: the timeliness of real-time analysis and decision-making, the accuracy of submicron-level structures, and the intelligence of auxiliary production decisions. Existing methods can be mainly divided into laser confocal and focused morphology recovery, among which: the laser confocal method scans the object to be measured layer by layer by focusing the laser beam, and collects the reflected signal to obtain the three-dimensional structure of the object; the focused morphology recovery method derives the three-dimensional morphology of the object to be measured through the focus dispersion caused by the inherent depth of field difference in the optical microscopy scanning process. Although the laser focusing method is more prominent in the accuracy of three-dimensional structure, it is limited by the cost of use, low reconstruction efficiency, information limitation and other problems. It is only suitable for scenes with partial accuracy and is not universal. The focused morphology recovery method has the fitting robustness of multi-scene adaptation, the application efficiency of large-scale reconstruction and the information sufficiency of morphological structure, which makes it have great application potential in the field of microscopic three-dimensional reconstruction.

[0004] However, most of the existing focus morphology recovery methods are modeled through the focus progression relationship or the relative relationship of defocus, which fails to fully combine the accuracy of focus progression and the relative robustness of defocus. As a result, the reconstruction schemes that rely on focus cues cannot cope with sparse depth of field sampling, which in turn leads to insufficient generalization of existing focus morphology recovery methods. In addition, unlike macroscopic optical camera lenses, microscopic optical microscopes have a narrow depth of field (about 0.2-120μm), and the robust identification of the focus area greatly affects the accuracy of the three-dimensional morphology. Therefore, how to efficiently couple the continuous relationship of focus and the relative relationship of defocus to improve the robustness of multiple scenes while ensuring the existing accuracy is one of the issues that need to be considered in the current microscopic three-dimensional morphology reconstruction.

[0005] From the above, we believe that in the process of microscopic three-dimensional morphology reconstruction with focused morphology recovery, the continuity of focus and the relativity of defocus are the core issues to effectively solve the accurate reconstruction of three-dimensional morphology. Therefore, this patent designs a feature construction module embedded with depth of field relative position encoding, and uses depth of field position encoding as prior information to strengthen the network's feature extraction of focus transition. The difference in performance of different position encodings ensures the effective judgment of the network. At the same time, this patent designs a multi-scale attention module for the specific weak depth of field problem in microscopic scenes, so as to enhance the edge judgment ability of the focused and defocused areas.

[0006] In general, the multi-scale microscopic three-dimensional morphology reconstruction method designed in this patent, which embeds depth of field relative position encoding, can be effectively generalized in different application scenarios based on the depth of field position prior and multi-scale focus representation, and improve the accuracy while ensuring the robustness of depth of field sampling. Summary of the invention

[0007] In order to overcome the problems of insufficient mining of relative relationship of focusing clues and single-scale judgment deviation of focusing area in existing microscopic 3D shape reconstruction, a multi-scale microscopic 3D shape reconstruction method with embedded depth of field relative position encoding is provided, which includes the following steps:

[0008] Step 1: Collect multi-depth image sequences in microscopic scenes n |1≤n≤N}, N represents the number of multi-depth image sequences, and the size of a single microscopic scene image is C×H×W, where H×W represents the height and width of a single image, and C represents the number of image channels. At the same time, the position coordinates of the multi-depth image sequence are obtained according to the position transformation of the microscope during the sampling process {L n |1≤n≤N};

[0009] Step 2: According to formula (1), a focus feature extraction module FM is constructed. The module includes a focus measurement module FM_conv with two sets of 3D convolutional layers, a 3D normalization layer and a ReLU activation layer, and a layer attention module FM_attention consisting of a 3D residual network and a 3D convolutional layer.

[0010] FM(x)=FM_conv(x)+FM_attention(x) (1)

[0011] Among them, x is the input tensor;

[0012] Step 3: For the multi-depth image sequence I obtained in step 1 n Substitute the focus feature extraction layer FM constructed in step 2 and obtain the focus feature matrix FM_M through formula (2):

[0013] FM_M=FM(I n ) (2)

[0014] Step 4: The position coordinates of the multi-depth image sequence obtained in step 1 {L n |1≤n≤N}, the depth position matrix L is obtained by the depth position encoding module FL_conv of formula (3) n _M,

[0015] L n _M=FL_conv(L n ) (3)

[0016] Then, the focusing feature matrix FM_M and the depth of field position matrix L are obtained by equation (4): n _M performs depth of field position embedding to obtain the focus-related feature FM_R,

[0017]

[0018] in It is the characteristic coupling mode;

[0019] Step 5: According to formula (5), a dual-branch parallel downsampling layer Down is constructed, which includes a 3D convolution downsampling layer and a 3D maximum pooling downsampling layer, and then the feature fusion of the above two layers is activated by the ReLU layer.

[0020] Down(x)=ReLU(Down_conv(x)+Down_maxpool(x)) (5)

[0021] Among them, x is the input tensor, the 3D convolution downsampling layer includes a 3D convolution layer and a 3D normalization layer, and the 3D maximum pooling downsampling layer includes a maximum pooling layer, a 3D convolution layer, and a 3D normalization layer;

[0022] Step 6: Perform multi-scale feature extraction on the focus correlation feature FM_R obtained in step 4 through formula (6) to obtain the aggregation scale feature FM_H.

[0023] FM_H=Hourglasses(FM_R) (6)

[0024] Among them, Hourglasses() is a multi-scale attention module, and its calculation method is shown in formula (7):

[0025] Hourglasses(x)=ConvTranspose3d(AvgPool3d(x))+x (7)

[0026] Among them, x is the input tensor, ConvTranspose3d() is a 3D deconvolution operation, and AvgPool3d() is a 3D average pooling operation;

[0027] Step 7: Iterate and repeat the above steps 5 and 6 twice to obtain the focused scale features FM_H′ and FM_H″ of two different scales;

[0028] Step 8: Perform 3D deconvolution operations on the focus scale feature set {FM_H″, FM_H′, FM_H} obtained in steps 6 and 7 according to equations (8) and (9) in sequence, and combine the adjacent focus scale features to obtain the final focus scale feature FM_C.

[0029] FM_T=ConvTranspose3d(FM_H″)+FM_H′ (8)

[0030] FM_C=ConvTranspose3d(FM_T)+FM_H (9)

[0031] Among them, ConvTranspose3d() represents 3D inverse convolution;

[0032] Step 9: The focused scale feature FM_C finally obtained in step 8 is mapped into a three-dimensional topography image D through 3D convolution according to formula (10).

[0033] D = Conv3d (FM_C) (10)

[0034] Among them, Conv3d() is a 3D convolutional layer.

[0035] Compared with the existing microscopic three-dimensional morphology reconstruction, the present invention has the following advantages:

[0036] (1) The depth position coding layer proposed in the present invention can effectively combine the focus position relationship between the depth image sequences, so as to adapt to the sampling bias existing under different sampling settings and can be effectively generalized to the sampling settings of different scenes.

[0037] (2) The multi-scale attention module proposed in the present invention can determine the focus and defocus range through multi-scale features, avoid the occurrence of depth holes, and effectively maintain robustness in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flowchart of a multi-scale microscopic three-dimensional shape reconstruction method with embedded depth-of-field relative position encoding;

[0039] Figure 2 A network structure diagram of a multi-scale microscopic three-dimensional shape reconstruction method embedded with depth-of-field relative position encoding;

[0040] Figure 3 A multi-depth image sequence of 10 images with different depths of field sampled in step 1 of embodiment 1 of the present invention

[0041] {I n |1≤n≤10}.

[0042] Figure 4 It is the three-dimensional morphology image D calculated in step 9 of Example 1 of the present invention. DETAILED DESCRIPTION

[0043] Example 1

[0044] like Figure 1 , Figure 2 , Figure 3 As shown, a multi-scale microscopic three-dimensional morphology reconstruction method embedded with depth of field relative position coding comprises the following steps:

[0045] Step 1: Collect multi-depth image sequences in microscopic scenes n |1≤n≤N}, N represents the number of multi-depth image sequences, the size of a single microscopic scene image is C×H×W, where H×W represents the height and width of a single image, H=256, W=256, C represents the number of image channels, C=3, and the position coordinates of the multi-depth image sequence are obtained according to the position transformation of the microscope during the sampling process {L n |1≤n≤N};

[0046] Step 2: According to formula (1), a focus feature extraction module FM is constructed. The module includes a focus measurement module FM_conv with two sets of 3D convolutional layers, a 3D normalization layer and a ReLU activation layer, and a layer attention module FM_attention consisting of a 3D residual network and a 3D convolutional layer.

[0047] FM(x)=FM_conv(x)+FM_attention(x) (1)

[0048] Among them, x is the input tensor;

[0049] Step 3: For the multi-depth image sequence I obtained in step 1 n Substitute the focus feature extraction layer FM constructed in step 2 and obtain the focus feature matrix FM_M through formula (2):

[0050] FM_M=FM(I n ) (2)

[0051] Step 4: The position coordinates of the multi-depth image sequence obtained in step 1 {L n |1≤n≤N}, the depth position matrix L is obtained by the depth position encoding module FL_conv of formula (3) n _M,

[0052] L n _M=FL_conv(L n) (3)

[0053] Then, the focusing feature matrix FM_M and the depth of field position matrix L are obtained by equation (4): n _M performs depth of field position embedding to obtain the focus-related feature FM_R,

[0054]

[0055] in It is the characteristic coupling mode;

[0056] Step 5: According to formula (5), a dual-branch parallel downsampling layer Down is constructed, which includes a 3D convolution downsampling layer and a 3D maximum pooling downsampling layer, and then the feature fusion of the above two layers is activated by the ReLU layer.

[0057] Down(x)=ReLU(Down_conv(x)+Down_maxpool(x)) (5)

[0058] Among them, x is the input tensor, the 3D convolution downsampling layer includes a 3D convolution layer and a 3D normalization layer, and the 3D maximum pooling downsampling layer includes a maximum pooling layer, a 3D convolution layer, and a 3D normalization layer;

[0059] Step 6: Perform multi-scale feature extraction on the focus correlation feature FM_R obtained in step 4 through formula (6) to obtain the aggregation scale feature FM_H.

[0060] FM_H=Hourglasses(FM_R) (6)

[0061] Among them, Hourglasses() is a multi-scale attention module, and its calculation method is shown in formula (7):

[0062] Hourglasses(x)=ConvTranspose3d(AvgPool3d(x))+x (7)

[0063] Among them, x is the input tensor, ConvTranspose3d() is a 3D deconvolution operation, and AvgPool3d() is a 3D average pooling operation;

[0064] Step 7: Iterate and repeat the above steps 5 and 6 twice to obtain the focused scale features FM_H′ and FM_H″ of two different scales;

[0065] Step 8: Perform 3D deconvolution operations on the focus scale feature set {FM_H″, FM_H′, FM_H} obtained in steps 6 and 7 according to equations (8) and (9) in sequence, and combine the adjacent focus scale features to obtain the final focus scale feature FM_C.

[0066] FM_T=ConvTranspose3d(FM_H″)+FM_H′ (8)

[0067] FM_C=ConvTranspose3d(FM_T)+FM_H (9)

[0068] Among them, ConvTranspose3d() represents 3D inverse convolution;

[0069] Step 9: Figure 4 As shown, the focused scale feature FM_C finally obtained in step 8 is mapped into a three-dimensional topography image D through 3D convolution according to formula (10), and its height and width are 256 and 256 respectively.

[0070] D = Conv3d (FM_C) (10)

[0071] Among them, Conv3d() is a 3D convolutional layer.

Claims

1. A multi-scale microscopic three-dimensional morphology reconstruction method embedded with depth of field relative position coding, characterized in that: The following steps are involved: Step 1: Collect multi-depth image sequences in microscopic scenes n |1≤n≤N}, N represents the number of multi-depth image sequences, and the size of a single microscopic scene image is C×H×W, where H×W represents the height and width of a single image, and C represents the number of image channels. At the same time, the position coordinates of the multi-depth image sequence are obtained according to the position transformation of the microscope during the sampling process {L n |1≤n≤N}; Step 2: According to formula (1), a focus feature extraction module FM is constructed. The module includes a focus measurement module FM_conv with two sets of 3D convolutional layers, a 3D normalization layer and a ReLU activation layer, and a layer attention module FM_attention consisting of a 3D residual network and a 3D convolutional layer. FM(x)=FM_conv(x)+FM_attention(x) (1) Where x is the input tensor; Step 3: For the multi-depth image sequence I obtained in step 1 n Substitute the focus feature extraction layer FM constructed in step 2 and obtain the focus feature matrix FM_M through formula (2): FM_M=FM(I n ) (2) Step 4: The position coordinates of the multi-depth image sequence obtained in step 1 {L n |1≤n≤N}, the depth position matrix L is obtained by the depth position encoding module FL_conv of formula (3) n _M, L n _M=FL_conv(L n ) (3) Then, the focusing feature matrix FM_M and the depth of field position matrix L are obtained by equation (4): n _M performs depth of field position embedding to obtain the focus-related feature FM_R, in It is the characteristic coupling mode; Step 5: According to formula (5), a dual-branch parallel downsampling layer Down is constructed, which includes a 3D convolution downsampling layer and a 3D maximum pooling downsampling layer, and then the feature fusion of the above two layers is activated by the ReLU layer. Down(x)=ReLU(Down_conv(x)+Down_maxpool(x)) (5) Among them, x is the input tensor, the 3D convolution downsampling layer includes a 3D convolution layer and a 3D normalization layer, and the 3D maximum pooling downsampling layer includes a maximum pooling layer, a 3D convolution layer, and a 3D normalization layer; Step 6: Perform multi-scale feature extraction on the focus correlation feature FM_R obtained in step 4 through formula (6) to obtain the aggregation scale feature FM_H. FM_H=Hourglasses(FM_R) (6) Among them, Hourglasses() is a multi-scale attention module, and its calculation method is shown in formula (7): Hourglasses(x)=ConvTranspose3d(AvgPool3d(x))+x (7) Among them, x is the input tensor, ConvTranspose3d() is a 3D deconvolution operation, and AvgPool3d() is a 3D average pooling operation; Step 7: Iterate and repeat the above steps 5 and 6 twice to obtain the focused scale features FM_H′ and FM_H″ of two different scales; Step 8: Perform 3D deconvolution operations on the focus scale feature set {FM_H″, FM_H′, FM_H} obtained in steps 6 and 7 according to equations (8) and (9) in sequence, and combine the adjacent focus scale features to obtain the final focus scale feature FM_C. FM_T=ConvTranspose3d(FM_H″)+FM_H′ (8) FM_C=ConvTranspose3d(FM_T)+FM_H (9) Among them, ConvTranspose3d() represents 3D inverse convolution; Step 9: The focused scale feature FM_C finally obtained in step 8 is mapped into a three-dimensional topography image D through 3D convolution according to formula (10). D = Conv3d (FM_C) (10) Among them, Conv3d() is a 3D convolutional layer.

Images