An Efficient Speckle Matching Method Based on Deep Learning

Abstract

The present invention discloses an efficient speckle matching method based on deep learning. First, a projector is used to project and a binocular stereo camera is used to synchronously collect speckle patterns. The collected speckle patterns are subjected to distortion and epipolar calibration using a circular plate calibration method. The calibrated patterns are used as network inputs. By combining an attention mechanism and a spatial pyramid pooling module (SPPM), the pattern features are extracted. The feature layers obtained by combining a series of convolutional layers are jointly used to construct a 4D cost volume. Cost aggregation is achieved through a multi-scale feature fusion method, and a disparity map is obtained through disparity regression. The disparity map is processed by the formula of stereo vision method to obtain a depth map, and finally, high-efficiency and high-precision three-dimensional imaging is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional imaging, and specifically relates to an efficient speckle matching method based on deep learning. Background Art

[0002] At present, the basic principle and related technical methods of the speckle projection technology have been relatively mature. The key technology affecting the speckle projection is a high-performance speckle stereo matching algorithm. However, due to the complex reflection characteristics of the surface of the object to be measured and the perspective difference between the two cameras, it is still difficult to ensure the global uniqueness of each pixel in the entire measurement space by projecting only one speckle pattern. There is a problem of poor measurement accuracy caused by incorrect matching during actual measurement. Therefore, the expensive computational overhead required for stereo matching poses a great challenge to potential applications based on real-time three-dimensional imaging. In addition, most of the current fast and high-precision three-dimensional imaging technologies still remain in the industrial application or even laboratory stage, and miniaturization and low cost make it difficult to popularize this technology to the consumer electronics field. Therefore, how to achieve fast and high-precision speckle stereo matching is gradually becoming its main development direction. Summary of the Invention

[0003] The object of the present invention is to propose an efficient speckle matching method based on deep learning.

[0004] The technical solution for achieving the object of the present invention is: an efficient speckle matching method based on deep learning, comprising the following steps:

[0005] Step 1: Use a projector to project onto the object to be measured, synchronously collect the speckle pattern through a binocular stereo camera, and perform distortion correction and epipolar calibration on the speckle pattern using the circular plate calibration method;

[0006] Step 2: Input the speckle pattern into the feature extraction sub-module of the network to obtain a feature tensor. The feature extraction sub-module includes two parallel parts and a fusion part for splicing the outputs of the two parallel parts. The first part of the two parallel parts is a spatial pyramid pooling module integrated with an attention mechanism, and the second part is several convolutional layers;

[0007] Step 3: Combine the feature tensor and the candidate disparity range to construct a 4D matching cost volume;

[0008] Step 4: Input the 4D matching cost volume into the cost aggregation module, achieve cost aggregation through the method of multi-scale feature fusion, and obtain a disparity map through disparity regression;

[0009] Step 5: Process the disparity map through the formula of stereo vision method to obtain a depth map and realize three-dimensional reconstruction.

[0010] Preferably, the spatial pyramid pooling module integrated with an attention mechanism is used to extract the pattern features. The specific process is as follows:

[0011] The speckle pattern passes through 5 convolutional layers with a stride of 2 to obtain a tensor of size H / 32×W / 32;

[0012] The tensor of H / 32×W / 32 undergoes 4 interpolation upsamplings to obtain a tensor of H / 2×W / 2;

[0013] The tensor of H / 2×W / 2 passes through 4 convolutional layers with a stride of 2 and 3 interpolation upsamplings to obtain a tensor of size 160×H / 4×W / 4.

[0014] Preferably, in the process of the speckle pattern passing through 5 convolutional layers with a stride of 2 to obtain a tensor of size H / 32×W / 32, the feature tensor processed by each convolutional layer is input into the activation function module. After being processed by the activation function, the weight information is connected to the feature tensor output by each convolutional layer to form a new feature map and input into the next convolutional layer. After the feature tensor output by the last convolutional layer is connected to the weight information, it undergoes 4 interpolation upsamplings to obtain a tensor of H / 2×W / 2.

[0015] Preferably, the activation function is specifically:

[0016] α = σ(F 2D (I(s)))

[0017] C o (s) = α×C i (s)

[0018] where, F 2D refers to the two-dimensional convolution operation, I(s) refers to the feature tensor obtained by processing the original image through the convolutional layer, σ refers to the activation function sigmoid, C i (s) refers to the initial cost volume before weight information processing, α refers to the weight information, and C o (s) refers to the concatenated cost volume obtained after weight information processing.

[0019] Preferably, the processing process of the second part of the feature extraction sub-module is: the images collected by the binocular stereo camera directly pass through two convolutional layers to obtain a tensor of size H / 4×W / 4.

[0020] Preferably, the fusion part concatenates two tensors of size 48×H / 4×W / 4 and two tensors of size 160×H / 4×W / 4 on the feature channels to obtain a tensor of size 256×H / 4×W / 4; after passing through two convolutional layers, a tensor of size 32×H / 2×W / 2 is obtained.

[0021] Preferably, the specific formula for constructing the 4D matching cost volume by combining the feature tensor and the candidate disparity range in step 3 is:

[0022] Cost(1:32, D i -D min +1, 1:H, 1:W - D i ) = Feature left (1:32, 1:H, 1:W - D i )

[0023] Cost(33:64, D i -D min +1, 1:H, 1:W - D i ) = Feature right (1:32, 1:H, D i :W)

[0024] Among them, Feature left and Feature right are feature tensors from two perspectives. Cost represents the cost quantity of the cost body. [D min , D max is the disparity range. D i is the candidate disparity. H×W is the size of the speckle pattern.

[0025] Preferably, the normalized probability of each candidate disparity Di in the four-dimensional cost body is obtained by using the softmax operation, and the predicted disparity map is obtained by weighted summation as shown in the following formula:

[0026]

[0027] Among them, [D min , D max is the disparity range. Softmax(·) represents the Softmax operation. Disparity represents the initial disparity map obtained by disparity regression. Cost is the 4D matching cost body after cost filtering.

[0028] Preferably, the formula of the volumetric vision method is:

[0029]

[0030] Among them, B represents the distance of the imaging system baseline, that is, the horizontal distance between the physical optical centers of the left and right cameras. f is the focal length of the two cameras. d is the horizontal disparity between two points of the object. Z refers to the depth information obtained by the stereoscopic vision method.

[0031] Compared with the prior art, the significant advantages of the present invention are: The speckle matching network proposed by the present invention can obtain a disparity map with higher accuracy and can predict the disparity map at a faster speed.

[0032] The following further describes the present invention in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Schematic diagram of the process of an efficient speckle matching method based on deep learning.

[0034] Figure 2 This is a diagram showing the basic principles of the speckle stereo matching method based on deep learning of the present invention. DETAILED DESCRIPTION

[0035] The concept of the present invention is: a high-efficiency speckle matching method based on deep learning. First, a projector is used to project and a binocular stereo camera synchronously collect speckle patterns. The collected speckle patterns are distorted and epipolar calibrated using a circular plate calibration method. The calibrated patterns are used as network input. The attention mechanism and spatial pyramid pooling module (SPPM) are combined to extract pattern features. The feature layers obtained by combining a series of convolutional layers are used to construct a 4-dimensional cost volume. Cost aggregation is achieved through multi-scale feature fusion. Finally, a disparity map is obtained through disparity regression. The disparity map is processed using a stereo vision formula to obtain a depth map, ultimately achieving high-efficiency and high-precision single-frame 3D imaging.

[0036] As an embodiment, an efficient speckle matching method based on deep learning is provided, wherein the specific steps are as follows:

[0037] Step 1: Use a projector to project onto the object to be measured, synchronously capture the speckle pattern with a binocular stereo camera, and use the circular plate calibration method to perform distortion correction and epipolar calibration on the speckle pattern;

[0038] Step 2: The calibrated speckle pattern is input into the feature extraction submodule of the stereo matching network. The feature extraction submodule consists of two parallel parts and a fusion part that splices the outputs of the two parallel parts. The two parallel parts include: one part extracts pattern features through the attention mechanism and the spatial pyramid pooling module (SPPM); the other part directly extracts features from the original image through a series of convolutional layers;

[0039] In the first part, five convolutional layers with a stride of 2 are used to obtain a tensor of size H / 32×W / 32; four interpolation upsampling layers are then used to obtain a tensor of size H / 2×W / 2.

[0040] The feature tensor obtained by each convolutional layer is input into the excitation function module. After being processed by the excitation function, the weight information is connected to the feature tensor output by each convolutional layer to form a new feature map input to the next convolutional layer. The feature tensor output by the last convolutional layer is connected with the weight information and then upsampled by 4 interpolations to obtain a tensor of H / 2×W / 2.

[0041] The tensor of H / 2×W / 2 passes through 4 convolutional layers with a stride of 2 and 3 interpolation upsamplings, and finally a tensor of size 160×H / 4×W / 4 is obtained;

[0042] In a further embodiment, the activation function is specifically:

[0043] α = σ(F 2D (I(s)))

[0044] C o (s) = α × C i (s)

[0045] where F 2D refers to a two-dimensional convolution operation, I(s) refers to the feature tensor obtained by processing the original image through the convolutional layer, σ refers to the activation function sigmoid, C i (s) refers to the initial cost volume before processing the weight information, α refers to the weight information, and C o (s) refers to the concatenated cost volume obtained after processing the weight information.

[0046] In the second part, the images collected by the binocular stereo camera are directly processed through two convolutional layers to obtain a tensor of size H / 4×W / 4;

[0047] Two tensors of size 48×H / 4×W / 4 and two tensors of size 160×H / 4×W / 4 are concatenated on the feature channels to obtain a tensor of size 256×H / 4×W / 4; after being processed through two convolutional layers, a tensor of size 32×H / 2×W / 2 is obtained.

[0048] Step 3: Combine the feature tensor and the candidate disparity range to construct a 4D matching cost volume, specifically:

[0049] Cost(1:32, D i -D min +1, 1:H, 1:W - D i ) = Feature left (1:32, 1:H, 1:W - D i )

[0050] Cost(33:64, D i -D min +1, 1:H, 1:W - D i ) = Feature right (1:32, 1:H, D i :W)

[0051] where Feature left and Feature rightThe feature tensors for two perspectives, and Cost represents the cost amount of the cost volume. [D min , D max is the disparity range, and D i is the candidate disparity.

[0052] Step 4: Use the 4D matching cost volume as the input of the cost aggregation module, implement cost aggregation through the method of multi-scale feature fusion, and obtain the disparity map through disparity regression. The specific process is as follows:

[0053] Use the softmax operation to obtain the normalized probability of each candidate disparity Di in the 4D cost volume, and perform weighted summation on it to obtain the predicted disparity map, as shown in the following formula:

[0054]

[0055] Among them, [D min , D max is the disparity range, Softmax(·) represents the Softmax operation, Disparity represents the initial disparity map obtained through disparity regression, Cost is the 4D matching cost volume after cost filtering, and the Softmax operation is as shown in the following formula:

[0056]

[0057] Use bilinear interpolation to obtain the initial disparity map at the original resolution.

[0058] Step 5: Process the disparity map into a depth map according to the formula of the stereo vision method, as shown in the following formula:

[0059]

[0060] Among them, B represents the distance of the baseline of the imaging system, that is, the horizontal distance between the physical optical centers of the left and right cameras, f is the focal length of the two cameras, and d is the horizontal disparity between two points of the object. Z refers to the depth information obtained through the stereo vision method.

[0061] The stereo matching network proposed by the present invention includes the following parts:

[0062] A feature network that extracts pattern features based on the attention mechanism and the spatial pyramid pooling module (SPPM);

[0063] Construct a 4D matching cost volume;

[0064] Use a three-dimensional convolutional layer to process the cost volume to achieve cost aggregation;

[0065] The 4D matching cost after cost aggregation obtains the disparity map through disparity regression;

[0066] Process the disparity map into a depth map according to the formula of the stereovision method.

[0067] Embodiment:

[0068] To verify the effectiveness of the present invention, a binocular stereovision system was built. The two cameras used in the binocular system of this embodiment are Basler industrial cameras (Basler acA640 750um), and the projector used is a digital projector (DLP4500 Pro). Using the content described in step 1, the projector projects and the binocular cameras synchronously collect the speckle pattern, and the pattern is corrected for distortion and epipolar calibration using the circular plate calibration method, and the obtained pattern is used as the network input. Figure 2 This is the basic principle diagram of the speckle stereo matching algorithm based on deep learning of the present invention. Using the content described in steps 2 to 5, high-efficiency and high-precision three-dimensional imaging is finally achieved.

Claims

1. An efficient speckle matching method based on deep learning, characterized in that, The method includes the following steps: Step 1: Project onto the object to be measured by a projector, synchronously collect the speckle pattern by a binocular stereo camera, and correct the distortion and calibrate the epipolar line of the speckle pattern by using a circular plate calibration method; Step 2: Input the speckle pattern into the feature extraction sub-module of the network to obtain a feature tensor. The feature extraction sub-module includes two parallel parts and a fusion part for splicing the outputs of the two parallel parts. The first part of the two parallel parts is a spatial pyramid pooling module incorporating an attention mechanism, and the second part is several convolutional layers; The spatial pyramid pooling module incorporating an attention mechanism is used to extract the pattern features. The specific process is as follows: The speckle pattern passes through 5 convolutional layers with a stride of 2 to obtain a tensor of size H / 32×W / 32; the tensor of size H / 32×W / 32 passes through 4 interpolation upsamplings to obtain a tensor of size H / 2×W / 2; The tensor of size H / 2×W / 2 passes through 4 convolutional layers with a stride of 2 and 3 interpolation upsamplings to obtain a tensor of size 160×H / 4×W / 4; In the process that the speckle pattern passes through 5 convolutional layers with a stride of 2 to obtain a tensor of size H / 32×W / 32, the feature tensor processed by each convolutional layer is input into the activation function module. After being processed by the activation function, the weight information is obtained and connected to the feature tensor output by each convolutional layer to form a new feature map, which is input into the next convolutional layer. The feature tensor output by the last convolutional layer is connected with the weight information and then passes through 4 interpolation upsamplings to obtain a tensor of size H / 2×W / 2; The activation function is specifically: α = σ(F 2D (I(s))) C o (s) = α × C i (s) Among them, F 2D refers to a two-dimensional convolution operation, I(s) refers to the feature tensor obtained by processing the original image through the convolution layer, σ refers to the activation function sigmoid, C i (s) refers to the initial cost volume before processing the weight information, α refers to the weight information, C o (s) refers to the concatenated cost volume obtained after processing the weight information; The processing process of the second part of the feature extraction sub-module is: the images collected by the binocular stereo camera directly pass through two convolutional layers to obtain a tensor of size H / 4×W / 4; The fusion part splices two tensors of size 48×H / 4×W / 4 and two tensors of size 160×H / 4×W / 4 on the feature channels to obtain a tensor of size 256×H / 4×W / 4; after being processed by two convolutional layers, a tensor of size 32×H / 2×W / 2 is obtained; Step 3: Construct a 4D matching cost volume by combining the feature tensor and the candidate disparity range; Step 4: Input the 4D matching cost volume into the cost aggregation module, realize cost aggregation by the method of multi-scale feature fusion, and obtain a disparity map through disparity regression; Step 5: Process the disparity map by the formula of stereo vision method to obtain a depth map and realize 3D reconstruction.

2. The high-efficiency speckle matching method based on deep learning according to claim 1, wherein The specific formula for constructing the 4D matching cost volume by combining the feature tensor and the candidate disparity range in Step 3 is: Cost(1:32,D i -D min +1,1:H,1:W-D i ) = Feature left (1:32,1:H,1:W-D i ) Cost(33:64,D i -D min +1,1:H,1:W-D i ) = Feature right (1:32,1:H,D i :W) Among them, Feature left and Feature right are feature tensors of two perspectives, Cost represents the cost amount of the cost body, [D min , D max is the disparity range, D i is the candidate disparity, and H×W is the size of the speckle pattern.

3. The efficient speckle matching method based on deep learning according to claim 1, wherein Use the softmax operation to obtain the normalized probability of each candidate disparity Di in the 4D matching cost volume, and perform weighted summation on it to obtain the predicted disparity map, as shown in the following formula: Among them, [D min , D max is the disparity range, Softmax(·) represents the Softmax operation, Disparity represents the initial disparity map obtained through disparity regression, and Cost is the 4D matching cost volume after cost filtering.

4. The efficient speckle matching method based on deep learning according to claim 1, wherein The formula of the stereo vision method is: Where B represents the distance of the imaging system baseline, that is, the horizontal distance between the physical optical centers of the left and right cameras, f is the focal length of the two cameras, d is the horizontal disparity between two points of the object, and Z refers to the depth information obtained by processing with the stereo vision method.

Images