Lightweight three-dimensional shape reconstruction method capable of flexibly adapting to scene

Abstract

The invention belongs to the field of three-dimensional shape reconstruction, and particularly relates to a light-weight three-dimensional shape reconstruction method capable of flexibly adapting to a scene. Comprising the following steps: firstly, acquiring an image sequence required by an experiment, and zooming the image sequence to obtain image sequences with different diffusion differences; then providing a multi-branch strategy with flexibly adapted resolution, extracting common characteristics of focusing dispersion measurement through resolution difference data, and further recording dispersion specificity of different resolutions; then, a neural network gradient confrontation phenomenon caused by resolution difference is relieved by adopting a guided distillation method, and it is ensured that high-resolution image input guides low-resolution image input; and finally, a U-shaped backbone network is adopted, a pyramid-shaped feature extraction mode is utilized, the feature dimension is reduced on the premise of ensuring the precision, and the light weight of the network structure is realized. The scene flexible adaptation method provided by the invention takes the resolution as an entry point, can effectively cope with the inherent dispersion phenomenon in the optical imaging process, and further can effectively improve the scene generalization of the three-dimensional morphology reconstruction method.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional shape reconstruction, and in particular relates to a lightweight three-dimensional shape reconstruction method with flexible scene adaptation. Background Art

[0002] The three-dimensional shape reconstruction technology aims to infer the three-dimensional information of the scene by means of optical imaging or laser ranging. It is an effective way to enable intelligent devices to perceive the world and human-computer interaction. It can serve the fields of precision parts processing, high-level intelligent driving and remote sensing terrain modeling. With the advent of information and intelligent industrial transformation, the existing three-dimensional shape reconstruction methods can no longer meet the needs of practical applications such as flexible decision-making in multiple scenarios and low-cost rapid reconstruction.

[0003] The existing three-dimensional reconstruction methods mainly include active projection and passive reception methods. The difference between the two is whether to apply additional detection signals to the scene to be measured. The active projection three-dimensional reconstruction method uses the detection components to actively project the distance measurement signal to the object to be measured, and analyzes the reflected signal to obtain the three-dimensional structure of the object to be measured; while the passive reception three-dimensional reconstruction method collects the information of the scene to be measured by inferring the distance association between scenes, which includes binocular measurement, focus transformation and multi-view scene matching. Among them, the active projection method has a strong dependence on equipment. While the measurement accuracy is high, the cost increases sharply. In addition, this type of method cannot obtain the original semantic information of the scene, such as color and structure. In addition, methods such as binocular measurement and multi-view scene matching are usually limited to specific modes or specific motion modes, and cannot achieve flexible three-dimensional structure reconstruction of the scene to be measured. In contrast, three-dimensional morphology reconstruction based on multi-focus image sequences can infer the three-dimensional structure of the scene to be measured by relying on the inherent depth of field information of the camera. This type of method has the advantages of simple and easy to obtain depth clues, high preservation of scene semantics, fast three-dimensional structure reconstruction, and extremely low equipment hardware cost.

[0004] Existing 3D shape reconstruction methods are mainly divided into two categories: mathematical modeling based on prior knowledge or data-driven deep modeling. Mathematical modeling methods based on prior knowledge usually start from the principle of optical imaging and supplemented by scene structure restoration, which cannot adapt to the complexity and variability brought by scene differences; data-driven deep modeling methods use labeled data to allow the neural network model to adaptively learn the depth clues of the scene, and combine the scene semantic information to infer the three-dimensional structure of the scene, which has the advantages of high precision and high efficiency.

[0005] Existing three-dimensional shape reconstruction methods use the evaluation mechanism in the unified focus-dispersion process, which cannot cope with the focus-dispersion deviation caused by the inherent circle of confusion phenomenon of the optical imaging system and the differences in scenes and equipment. Focus-dispersion strengthens the stability of depth cues while carrying the specificity of focus-shaped imaging. Although data-driven deep modeling methods bridge the above differences through large-scale high-precision data, data collection and information annotation will result in high labor costs. At the same time, the portability and easy deployment of three-dimensional measurement equipment require low-cost and high-efficiency measurement for three-dimensional shape reconstruction. Therefore, how to eliminate the focus-dispersion deviation of multi-depth images of different devices is the key to ensuring the high efficiency and high precision of three-dimensional measurement equipment.

[0006] As can be seen from the above, we believe that focus-dispersion deviation is the core problem of robust 3D shape reconstruction. Therefore, this patent proposes a multi-branch strategy with flexible resolution adaptation to address the problem of large performance differences in image inputs at different resolutions in existing methods. First, the common features of focus-dispersion measurements are extracted through resolution difference data to record the dispersion specificity of different resolutions. In addition, the guided distillation method is used to alleviate the neural network gradient confrontation phenomenon caused by resolution differences, and ensure that high-resolution image input guides low-resolution image input; at the same time, this patent adopts a U-shaped backbone network and uses a pyramid-shaped feature extraction method to reduce feature dimensions while ensuring accuracy, meeting lightweight requirements. Summary of the invention

[0007] In order to overcome the limitations of existing three-dimensional shape reconstruction methods in dealing with optical aberrations and resolution problems, the present invention provides a lightweight three-dimensional shape reconstruction method with flexible scene adaptation, comprising the following steps:

[0008] Step 1: Collect the image sequence required for the experiment {L n |1≤n≤N}, where N is the number of image sequences, the size of a single image is C×H×W, H×W represents the height and width of the image, and C is the number of image channels;

[0009] Step 2: According to formula (1), a focus feature extraction module FeaExt for focus stack is constructed.

[0010] FeaExt(x)=FeaExt_I(x)+FeaExt_SR(x) (1)

[0011] Among them, FeaExt_I is the initial feature extraction module, which mainly uses dilated convolution and feature pyramid for information extraction, FeaExt_SR is the focused edge detection module, which is mainly composed of 3D convolution, and x is the input tensor;

[0012] Step 3: construct the effective downsampling module EDown according to formula (2):

[0013] EDown(x)=3DPooling(x) (2)

[0014] Among them, this module uses 3D convolution maximum pooling 3DPooling as a downsampling operation to extract the representative value of the focus area locally and effectively transfer the focus feature. x is the input tensor;

[0015] Step 4: The image sequence L acquired in step 1 n Input the feature extraction module FeaExt and the effective downsampling module EDown constructed in step 2 and step 3, and then obtain the focused feature matrix FE_M through equations (3) and (4).

[0016] FE_M′=FeaExt(L n ) (3)

[0017]

[0018] in, It is the characteristic coupling mode;

[0019] Step 5: According to formula (5), a multi-branch structure Res with flexible adaptation to image resolution is constructed. The structure consists of a 3D convolution layer Res_3DConv including a convolution layer and a normalization layer, and a 3D maximum pooling layer Res_3DMaxpool consisting of a maximum pooling layer, a convolution layer and a normalization layer. The features of the above two layers are fused through the ReLU activation function.

[0020] Res(x)=ReLU(Res_3DConv(x)+Res_3DMaxpool(x)) (5)

[0021] Where x is the input tensor;

[0022] Step 6: extract multi-scale information from the focus correlation feature FE_M obtained in step 4 according to formula (6), fuse features of different scales, and obtain the aggregated scale feature FE_H.

[0023] FE_H=DeConv3d(AvgPool3d(FE_M))+FE_M (6)

[0024] Among them, DeConv3d represents 3D deconvolution, and AvgPool3d is a 3D average pooling operation;

[0025] Step 7: Repeat steps 5 and 6 twice for the clustered scale feature FE_H obtained in step 6 to obtain a focused scale feature set {FE_H″, FE_H′, FE_H} of different resolutions. Perform a 3D deconvolution operation according to formula (7) to obtain the scale information FE_T.

[0026] FE_T=DeConv3d(FE_H″)+FE_H′ (7)

[0027] Among them, DeConv3d represents 3D deconvolution;

[0028] Step 8: Perform 3D deconvolution again on the scale information FE_T obtained in step 7 according to formula (8), and combine it with adjacent focused scale features to obtain the focused scale feature FE_C.

[0029] FE_C=DeConv3d(FE_T)+FE_H (8)

[0030] Among them, DeConv3d represents 3D deconvolution;

[0031] Step 9: According to formula (9), the scale information of the image data FE_C is adjusted by bilinear interpolation to match the corresponding focal stack, and FE_C is smoothed by the Softplus function.

[0032] FE_C=Softplus(FE_C)+10 -6 (9)

[0033] Then, FeaExt_C is normalized according to formula (10):

[0034]

[0035] Where N is the total number of samples in the focal stack. Finally, the depth estimate is calculated by weighted summing of the image sequence using equation (11):

[0036]

[0037] Where N is the total number of samples in the focus stack, FD is the focus position information of the corresponding image, It is the characteristic coupling mode;

[0038] Step 10: The focus scale feature FE_C finally obtained in step 9 is mapped into a three-dimensional topography image DfF through 3D convolution according to formula (12).

[0039] DfF=3DConv(FE_C) (12)

[0040] Among them, 3DConv is a 3D convolution operation;

[0041] Step 11, use the loss function of formula (13) to guide the distillation process, and use the prediction of the high-resolution branch to provide auxiliary supervision signals for the prediction of the low-resolution branch.

[0042]

[0043] Among them, DfF_H is the prediction of the high-resolution branch, and DfF_M is the prediction result of the low-resolution branch.

[0044] Compared with the existing three-dimensional shape reconstruction method, this patent has the following advantages:

[0045] (1) This patent proposes a multi-branch strategy with flexible resolution adaptation. The common features of focused diffusion measurement are extracted through resolution difference data to record the diffusion specificity of different resolutions; in addition, the guided distillation method is used to alleviate the neural network gradient confrontation phenomenon caused by resolution difference, ensuring that high-resolution image input guides low-resolution image input.

[0046] (2) This patent adopts a U-shaped backbone network and a pyramid-shaped feature extraction method to reduce the feature dimension while ensuring accuracy, meeting the lightweight requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A flowchart of a lightweight 3D shape reconstruction method that is flexible and adaptable to different scenes;

[0048] Figure 2 A network structure diagram of a lightweight 3D shape reconstruction method that can be flexibly adapted to different scenarios;

[0049] Figure 3 10 samples of the test data image sequence in step 1 of the present invention;

[0050] Figure 4 The three-dimensional morphology image with a resolution of 512×512 finally calculated in Example 1 of the present invention;

[0051] Figure 5 This is a three-dimensional morphology image with a resolution of 256×256 finally calculated in Example 1 of the present invention. DETAILED DESCRIPTION

[0052] Example 1

[0053] like Figure 1 , Figure 2 As shown, a lightweight three-dimensional shape reconstruction method with flexible scene adaptation includes the following steps:

[0054] Step 1: Collect the image sequence required for the experiment n |1≤n≤N}, where N is the number of image sequences, the size of a single image is C×H×W, H×W represents the height and width of the image, C is the number of image channels, C=10, such as Figure 3 As shown;

[0055] Step 2: According to formula (1), a focus feature extraction module FeaExt for focus stack is constructed.

[0056] FeaExt(x)=FeaExt_I(x)+FeaExt_SR(x) (1)

[0057] Among them, FeaExt_I is the initial feature extraction module, which mainly uses dilated convolution and feature pyramid for information extraction, FeaExt_SR is the focused edge detection module, which is mainly composed of 3D convolution, and x is the input tensor;

[0058] Step 3: construct the effective downsampling module EDown according to formula (2):

[0059] EDown(x)=3DPooling(x) (2)

[0060] Among them, this module uses 3D convolution maximum pooling 3DPooling as a downsampling operation to extract the representative value of the focus area locally and effectively transfer the focus feature. x is the input tensor;

[0061] Step 4: The image sequence L acquired in step 1 n Input the feature extraction module FeaExt and the effective downsampling module EDown constructed in step 2 and step 3, and then obtain the focused feature matrix FE_M through equations (3) and (4).

[0062] FE_M′=FeaExt(L n ) (3)

[0063]

[0064] in, It is the characteristic coupling mode;

[0065] Step 5: According to formula (5), a multi-branch structure Res with flexible adaptation to image resolution is constructed. The structure consists of a 3D convolution layer Res_3DConv including a convolution layer and a normalization layer, and a 3D maximum pooling layer Res_3DMaxpool consisting of a maximum pooling layer, a convolution layer and a normalization layer. The features of the above two layers are fused through the ReLU activation function.

[0066] Res(x)=ReLU(Res_3DConv(x)+Res_3DMaxpool(x)) (5)

[0067] Where x is the input tensor;

[0068] Step 6: extract multi-scale information from the focus correlation feature FE_M obtained in step 4 according to formula (6), fuse features of different scales, and obtain the aggregated scale feature FE_H.

[0069] FE_H=DeConv3d(AvgPool3d(FE_M))+FE_M (6)

[0070] Among them, DeConv3d represents 3D deconvolution, and AvgPool3d is a 3D average pooling operation;

[0071] Step 7: Repeat steps 5 and 6 twice for the clustered scale feature FE_H obtained in step 6 to obtain a focused scale feature set {FE_H″, FE_H′, FE_H} of different resolutions. Perform a 3D deconvolution operation according to formula (7) to obtain the scale information FE_T.

[0072] FE_T=DeConv3d(FE_H″)+FE_H′ (7)

[0073] Among them, DeConv3d represents 3D deconvolution;

[0074] Step 8: Perform 3D deconvolution again on the scale information FE_T obtained in step 7 according to formula (8), and combine it with adjacent focused scale features to obtain the focused scale feature FE_C.

[0075] FE_C=DeConv3d(FE_T)+FE_H (8)

[0076] Among them, DeConv3d represents 3D deconvolution;

[0077] Step 9: According to formula (9), the scale information of the image data FE_C is adjusted by bilinear interpolation to match the corresponding focal stack, and FE_C is smoothed by the Softplus function.

[0078] FE_C=Softplus(FE_C)+10 -6 (9)

[0079] Then, FeaExt_C is normalized according to formula (10):

[0080]

[0081] Where N is the total number of samples in the focal stack. Finally, the depth estimate is calculated by weighted summing of the image sequence using equation (11):

[0082]

[0083] Where N is the total number of samples in the focus stack, FD is the focus position information of the corresponding image, It is the characteristic coupling mode;

[0084] Step 10: The focus scale feature FE_C finally obtained in step 9 is mapped into a three-dimensional topography image DfF through 3D convolution according to formula (12).

[0085] DfF=3DConv(FE_C) (12)

[0086] Among them, 3DConv is a 3D convolution operation;

[0087] Step 11, use the loss function of formula (13) to guide the distillation process, and use the prediction of the high-resolution branch to provide auxiliary supervision signals for the prediction of the low-resolution branch.

[0088]

[0089] in, Figure 4 DfF_H is the high-resolution branch prediction, Figure 5 DfF_M is the prediction result of the low-resolution branch.

Claims

1. A lightweight three-dimensional shape reconstruction method with flexible scene adaptation, comprising the following steps: Step 1: Collect the image sequence required for the experiment {L n |1≤n≤N}, where N is the number of image sequences, the size of a single image is C×H×W, H×W represents the height and width of the image, and C is the number of image channels; Step 2: According to formula (1), a focus feature extraction module FeaExt for focus stack is constructed. FeaExt(x)=FeaExt_I(x)+FeaExt_SR(x) (1) Among them, FeaExt_I is the initial feature extraction module, which mainly uses dilated convolution and feature pyramid for information extraction, FeaExt_SR is the focused edge detection module, which is mainly composed of 3D convolution, and x is the input tensor; Step 3: construct the effective downsampling module EDown according to formula (2): EDown(x)=3DPooling(x) (2) Among them, this module uses 3D convolution maximum pooling 3DPooling as a downsampling operation to extract the representative value of the focus area locally and effectively transfer the focus feature. x is the input tensor; Step 4: The image sequence L acquired in step 1 n Input the feature extraction module FeaExt and the effective downsampling module EDown constructed in step 2 and step 3, and then obtain the focused feature matrix FE_M through equations (3) and (4). FE_M′=FeaExt(L n ) (3) in, It is the characteristic coupling mode; Step 5: According to formula (5), a multi-branch structure Res with flexible adaptation to image resolution is constructed. The structure consists of a 3D convolution layer Res_3DConv including a convolution layer and a normalization layer, and a 3D maximum pooling layer Res_3DMaxpool consisting of a maximum pooling layer, a convolution layer and a normalization layer. The features of the above two layers are fused through the ReLU activation function. Res(x)=ReLU(Res_3DConv(x)+Res_3DMaxpool(x)) (5) Where x is the input tensor; Step 6: extract multi-scale information from the focus correlation feature FE_M obtained in step 4 according to formula (6), fuse features of different scales, and obtain the aggregated scale feature FE_H. FE_H=DeConv3d(AvgPool3d(FE_M))+FE_M (6) Among them, DeConv3d represents 3D deconvolution, and AvgPool3d is a 3D average pooling operation; Step 7: Repeat steps 5 and 6 twice for the clustered scale feature FE_H obtained in step 6 to obtain a focused scale feature set {FE_H″, FE_H′, FE_H} of different resolutions. Perform a 3D deconvolution operation according to formula (7) to obtain the scale information FE_T. FE_T=DeConv3d(FE_H″)+FE_H′ (7) Among them, DeConv3d represents 3D deconvolution; Step 8: Perform 3D deconvolution again on the scale information FE_T obtained in step 7 according to formula (8), and combine it with adjacent focused scale features to obtain the focused scale feature FE_C. FE_C=DeConv3d(FE_T)+FE_H (8) Among them, DeConv3d represents 3D deconvolution; Step 9: According to formula (9), the scale information of the image data FE_C is adjusted by bilinear interpolation to match the corresponding focal stack, and FE_C is smoothed by the Softplus function. FE_C=Softplus(FE_C)+10 -6 (9) Then, FeaExt_C is normalized according to formula (10): Where N is the total number of samples in the focal stack. Finally, the depth estimate is calculated by weighted summing of the image sequence using equation (11): Where N is the total number of samples in the focus stack, FD is the focus position information of the corresponding image, It is the characteristic coupling mode; Step 10: The focus scale feature FE_C finally obtained in step 9 is mapped into a three-dimensional topography image DfF through 3D convolution according to formula (12). DfF=3DConv(FE_C) (12) Among them, 3DConv is a 3D convolution operation; Step 11, use the loss function of formula (13) to guide the distillation process, and use the prediction of the high-resolution branch to provide auxiliary supervision signals for the prediction of the low-resolution branch. Among them, DfF_H is the prediction of the high-resolution branch, and DfF_M is the prediction result of the low-resolution branch.

Images