Lightweight three-dimensional morphology reconstruction method based on time-frequency hybrid modulation

Abstract

The invention belongs to the technical field of microscopic three-dimensional morphology reconstruction in the field of computer vision, and particularly relates to a lightweight three-dimensional morphology reconstruction method based on time-frequency hybrid modulation. The method comprises the following steps: performing equal-interval extraction on image sequences of different focal planes acquired in a microscopic scene, extracting different features by using a multi-scale perceptron, inputting the features into a grouping parallel convolution module to respectively obtain local extreme points so as to obtain an initial feature set of the image sequences, and finally summarizing each group of focusing information through point-by-point convolution so as to obtain an initial feature set of the image sequences; and selecting the maximum focusing value in the whole domain to obtain a high-precision three-dimensional shape reconstruction result of the microscopic scene. According to the method, the parameter quantity of a network model and the reasoning time of an acceleration model can be reduced at the same time, and the real-time reconstruction problem of a microscopic scene is effectively solved.

Description

Technical Field

[0001] The invention relates to a three-dimensional shape reconstruction method in the field of computer vision, and in particular to a lightweight three-dimensional shape reconstruction method based on time-frequency hybrid modulation. Background Art

[0002] Recovering depth using a single RGB camera is important for 3D vision applications such as 3D reconstruction, biomedical observation, and quality control. Existing works have explored various cues to address this issue, such as ego-motion estimation and structured light patterns, but they require additional motion or equipment, which limits their application on smart devices and handheld cameras. In contrast, focus cues are easily obtained from the focus stack generated by scanning the focal plane of the camera to obtain a well-focused image. With fixed focal length and aperture size, the relative distance between the object and the camera determines the focus area. Focus depth aims to estimate depth by analyzing the focus information on a set of images with gradually changing optical focus.

[0003] With the advent of convolutional neural networks, many methods use them to extract meaningful image features. For example, at present, there are three main methods for 3D shape reconstruction based on convolutional neural networks: DDFF, AiFDepthNet, and DfFintheWild. DDFF solved the problem of recovering depth from focal information for the first time with an end-to-end learning method. The main advantage of this network model is that it can generate output at the pixel level, which is more prominent than traditional model design methods. Traditional model design usually relies on regularization techniques to obtain meaningful depth maps, but this processing method usually results in overly smooth output results. In contrast, the deep learning-based DDFF method combines a pixel-by-pixel focus strategy to more accurately obtain the depth information of each pixel. AiFDepthNet proposes a method for inferring depth maps and all-focus images from a set of focal stacks. The network adopts a shared architecture to effectively capture the relationship between depth and all-focus estimation, and simultaneously estimates depth and all-focus images from the input focal depth sequence. Therefore, this method can be supervised by using the ground truth depth for training, or use all-focus images as supervisory signals for unsupervised training. The success of the DfFintheWild method can be attributed to the following features: a new end-to-end inference framework that simplifies the depth estimation process to the greatest extent and enhances the network's ability to produce accurate depth predictions in the presence of alignment complexity; by introducing a sharp region detection module, which effectively resolves the blurriness caused by subtle focus changes in weak texture areas; including an efficient downsampling module that enhances the extraction of focus information during feature extraction. This module not only retains the original feature representation, but also facilitates the extraction of informative features from the focused area. Through optimized downsampling, the network is able to capture subtle changes in blur between input images, thereby improving depth estimation performance.

[0004] However, existing work has analyzed the spectral filtering of multi-head self-attention and convolutional neural networks and found that multi-head self-attention is more like a low-pass filter, while convolutional neural networks are more suitable for high-pass filters. Combining the two in a specific sequence can improve model performance. Therefore, by enhancing the capture of high-frequency components by multi-head self-attention, the performance of the multi-head self-attention-based model is improved. On the contrary, we believe that improving the low-pass filter capability of convolution can also improve the performance of the model. The above three methods are all built based on convolutional neural networks, and we compare and analyze the depth maps inferred by the three methods with the true labels, and find that the inference results are often accompanied by blurred contours and discontinuous depth information. We then perform spectral analysis on the inference results and the true labels. Compared with the spectrum of the true labels, the results inferred by these three methods obviously lack more low-frequency information.

[0005] Through the above research status analysis, we believe that the existing three-dimensional morphology reconstruction based on deep learning multi-focus images has the following shortcomings: the existing methods are mainly built based on convolutional networks and lack the extraction of low-frequency information; although the hybrid network of self-attention and convolution can solve this problem, the square complexity of self-attention is obviously too energy-consuming for high-resolution images in real scenes.

[0006] In summary, we believe that in multi-focus 3D shape reconstruction based on convolutional neural networks, it is necessary to balance computational complexity, memory consumption and accuracy while making full use of high and low frequency information in the feature extraction stage. Therefore, in the process of building the network model, this patent enhances the extraction of low frequency information while also ensuring low computational complexity, low memory consumption and higher accuracy. By designing a new frequency mixer, the multi-focus 3D shape reconstruction method can be made more lightweight while making better use of high and low frequency information. Summary of the invention

[0007] In order to overcome the shortcomings of the existing solutions, the present invention aims to provide a lightweight three-dimensional shape reconstruction method based on time-frequency hybrid modulation, comprising the following steps:

[0008] Step 1: Multi-focus image sequence of a given scene Where N represents the total number of image sequences, n represents the position of the image sequence, and its value range is 1≤n≤N. According to formula (1), the output visual feature X is obtained c ,

[0009]

[0010] Where F() is the feature extractor;

[0011] Step 2: The visual feature X obtained in step 1c According to formula (2), fast Fourier transform is first used, and then frequency shift operation is performed to obtain the feature X where the low-frequency component is located at the center of the spectrum. c ,

[0012] X c =fftshift(FFT(X c )) (2)

[0013] Where FFT() stands for Fast Fourier Transform, and fftshift() stands for frequency shift operation;

[0014] Step 3: The feature X obtained in step 2 c According to formula (3), we can get the low-frequency subset

[0015]

[0016] Among them, M lf represents a binary mask, and the expression is shown in formula (4).

[0017]

[0018] Where (h, w) represents the position coordinates of a point in the frequency domain, A lf represents the radius;

[0019] Step 4: For the low-frequency subset obtained in step 3 According to formula (5), the low-frequency feature X is obtained by using the learned shared weights c ,

[0020]

[0021] Where W is from The complex weight matrix learned in , ⊙ represents the Hadamard dot product;

[0022] Step 5: For the low-frequency feature X obtained in step 4 c According to equation (6), the inverse Fourier transform is converted to the spatial domain.

[0023]

[0024] Where iFFT() stands for inverse Fourier transform, is the final low-frequency feature output;

[0025] Step 6: The visual feature X obtained in step 1 c According to formula (7), four group convolutions with different convolution kernel sizes are used.

[0026]

[0027] Where G4 means the number of groups is 4, Conv3D() means the convolution operation, and s means different convolution kernel sizes;

[0028] Step 7: For the characteristic components obtained in steps 5 and 6 and According to formula (8), we can get X 1 ,

[0029]

[0030] Where Fusion() means fusing features of different frequencies;

[0031] Step 8: The feature set X obtained in step 7 1 According to formula (9), iterative calculation is used to obtain the feature set X 2 and X 3 ,

[0032] X n =Circulat(Downsample(X n-1 )),2≤n≤3 (9)

[0033] Circulat() means two-cycle operation, and Downsample() means downsampling operation;

[0034] Step 9: For the feature set X obtained in step 8 3 According to formula (10), after multi-scale decomposition and fusion, the fusion feature X is obtained fusion ,

[0035] X fusion =MsFusion(X 3 ) (10)

[0036] Where MsFusion() represents a multi-scale feature extractor;

[0037] Step 10: The fused feature set X obtained in step 9 fusion According to formula (11), the aggregation feature

[0038]

[0039] Where [,] represents concatenation, Aggre() represents feature aggregation, and Interp() represents the bilinear interpolation function;

[0040] Step 11: aggregate features obtained in step 10 According to formula (12), the predicted depth map D is obtained by normalization.

[0041]

[0042] Where softplus() is the normalization function.

[0043] Compared with the prior art, the present invention has the following advantages:

[0044] (1) The lightweight 3D shape reconstruction method proposed in this paper makes full use of the relationship between focal stack and depth, which can effectively improve the accuracy of 3D structure estimation of different types of scenes;

[0045] (2) The lightweight three-dimensional morphology reconstruction method proposed in the present invention has the advantages of low cost and fast reasoning, and can be applied to the field of industrial three-dimensional measurement with high timeliness requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flowchart of a lightweight 3D shape reconstruction method based on time-frequency hybrid modulation;

[0047] Figure 2 A network diagram of a lightweight 3D shape reconstruction method based on time-frequency hybrid modulation;

[0048] Figure 3 is the real scene focus stack given in step 1 in embodiment 1 of the present invention;

[0049] Figure 4 The depth map D of the real scene is calculated in step 11 in embodiment 1 of the present invention. DETAILED DESCRIPTION

[0050] Example 1

[0051] like Figure 1 and Figure 2 As shown, a lightweight three-dimensional morphology reconstruction method based on time-frequency hybrid modulation includes the following steps:

[0052] Step 1: Multi-focus image sequence of a given scene like Figure 3 As shown in the figure, N represents the total number of image sequences, n represents the position of the image sequence, and its value range is 1≤n≤N. According to formula (1), the output visual feature X is obtained c ,

[0053]

[0054] Where F() is the feature extractor;

[0055] Step 2: The visual feature X obtained in step 1 c According to formula (2), fast Fourier transform is first used, and then frequency shift operation is performed to obtain the feature X where the low-frequency component is located at the center of the spectrum.c ,

[0056] X c =fftshift(FFT(X c )) (2)

[0057] Where FFT() stands for Fast Fourier Transform, and fftshift() stands for frequency shift operation;

[0058] Step 3: The feature X obtained in step 2 c According to formula (3), we can get the low-frequency subset

[0059]

[0060] Among them, M lf represents a binary mask, and the expression is shown in formula (4).

[0061]

[0062] Where (h, w) represents the position coordinates of a point in the frequency domain, A lf represents the radius;

[0063] Step 4: For the low-frequency subset obtained in step 3 According to formula (5), the low-frequency feature X is obtained by using the learned shared weights c ,

[0064]

[0065] Where W is from The complex weight matrix learned in , ⊙ represents the Hadamard dot product;

[0066] Step 5: For the low-frequency feature X obtained in step 4 c According to equation (6), the inverse Fourier transform is converted to the spatial domain.

[0067]

[0068] Where iFFT() stands for inverse Fourier transform, is the final low-frequency feature output;

[0069] Step 6: The visual feature X obtained in step 1 c According to formula (7), four group convolutions with different convolution kernel sizes are used.

[0070]

[0071] Where G4 means the number of groups is 4, Conv3D() means the convolution operation, and s means different convolution kernel sizes;

[0072] Step 7: For the characteristic components obtained in steps 5 and 6 and According to formula (8), we can get X 1 ,

[0073]

[0074] Where Fusion() means fusing features of different frequencies;

[0075] Step 8: The feature set X obtained in step 7 1 According to formula (9), iterative calculation is used to obtain the feature set X 2 and X 3 ,

[0076] X n =Circulat(Downsample(X n-1 )),2≤n≤3 (9)

[0077] Circulat() means two-cycle operation, and Downsample() means downsampling operation;

[0078] Step 9: For the feature set X obtained in step 8 3 According to formula (10), after multi-scale decomposition and fusion, the fusion feature X is obtained fusion ,

[0079] X fusion =MsFusion(X 3 ) (10)

[0080] Where MsFusion() represents a multi-scale feature extractor;

[0081] Step 10: The fused feature set X obtained in step 9 fusion According to formula (11), the aggregation feature

[0082]

[0083] Where [,] represents concatenation, Aggre() represents feature aggregation, and Interp() represents the bilinear interpolation function;

[0084] Step 11: aggregate features obtained in step 10 Normalize according to formula (12) to obtain the predicted depth map D, as Figure 4 As shown,

[0085]

[0086] Where softplus() is the normalization function.

Claims

1. A lightweight three-dimensional shape reconstruction method based on time-frequency hybrid modulation, characterized in that: The following steps are involved: Step 1: Multi-focus image sequence of a given scene Where N represents the total number of image sequences, n represents the position of the image sequence, and its value range is 1≤n≤N. According to formula (1), the output visual feature X is obtained c , Where F() is the feature extractor; Step 2: The visual feature X obtained in step 1 c According to formula (2), fast Fourier transform is first used, and then frequency shift operation is performed to obtain the feature X where the low-frequency component is located at the center of the spectrum. c , X c =fftshift(FFT(X c )) (2) Where FFT() stands for Fast Fourier Transform, and fftshift() stands for frequency shift operation; Step 3: The feature X obtained in step 2 c According to formula (3), we can get the low-frequency subset Among them, M lf represents a binary mask, and the expression is shown in formula (4). Where (h, w) represents the position coordinates of a point in the frequency domain, A lf represents the radius; Step 4: For the low-frequency subset obtained in step 3 According to formula (5), the low-frequency feature X is obtained by using the learned shared weights c , Where W is from The complex weight matrix learned in , ⊙ represents the Hadamard dot product; Step 5: For the low-frequency feature X obtained in step 4 c According to equation (6), the inverse Fourier transform is converted to the spatial domain. Where iFFT() stands for inverse Fourier transform, is the final low-frequency feature output; Step 6: The visual feature X obtained in step 1 c According to formula (7), four group convolutions with different convolution kernel sizes are used. Where G4 means the number of groups is 4, Conv3D() means the convolution operation, and s means different convolution kernel sizes; Step 7: For the characteristic components obtained in steps 5 and 6 and According to formula (8), we can get X 1 , Where Fusion() means fusing features of different frequencies; Step 8: The feature set X obtained in step 7 1 According to formula (9), iterative calculation is used to obtain the feature set X 2 and X 3 , X n =Circulated(Downsample(X n-1 )),2≤n≤3 (9) Circulat() means two-cycle operation, and Downsample() means downsampling operation; Step 9: For the feature set X obtained in step 8 3 According to formula (10), after multi-scale decomposition and fusion, the fusion feature X is obtained fusion , X fusion =MsFusion(X 3 ) (10) Where MsFusion() represents a multi-scale feature extractor; Step 10: The fused feature set X obtained in step 9 fusion According to formula (11), the aggregation feature Where [,] represents concatenation, Aggre() represents feature aggregation, and Interp() represents the bilinear interpolation function; Step 11: aggregate features obtained in step 10 According to formula (12), the predicted depth map D is obtained by normalization. Where softplus() is the normalization function.

Images