A multi-view depth estimation method

Abstract

The present invention provides a multi-view depth estimation method, which has the following specific steps: Step 1: Image input: Obtain N+1 different images through the camera, and the front view image and the images in the other directions are respectively used as reference images I i＝0 and target image I i , the value range of i is 0 to N; Step 2: Feature extraction: It includes two parts: FPN network module and CA module; Step 2.1: The FPN network module extracts initial feature maps of different scales based on the image obtained in Step 1; Step 2.2: After the FPN network module extracts features of the input camera image at three different scales, it can solve the existing problems of inaccurate or unpredictable depth information predicted by traditional depth estimation methods due to occlusion problems, weak texture areas, reflective surfaces, and repeated patterns in the collected images.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and deep learning technology, and specifically relates to a multi-view depth estimation method. Background Art

[0002] Obtaining depth information from images is crucial in numerous fields, including autonomous driving, industrial inspection, healthcare, aerospace, and 3D reconstruction. Multi-view depth map recovery holds great promise. Specifically, multi-view depth estimation methods use multiple images of an object or scene captured by a camera from different angles as the primary input for depth estimation. Computer vision algorithms are then used to generate a depth map.

[0003] A depth map is an image that uses the distance from a point in the real scene to the camera as its pixel value. Smaller depth values indicate closer points in the real scene are located to the camera. Acquiring depth information from a target scene can be categorized into two methods: active and passive. Active depth acquisition utilizes highly precise and sophisticated hardware, such as LiDAR (LiDAR), which emits laser light and uses sophisticated Time of Flight (TOF) ranging technology to determine the distance from the target object to the camera, thereby obtaining depth information. While active methods are faster, more convenient, and more accurate, these devices are generally expensive, resulting in a high cost for depth acquisition. They also have requirements for the scene's external environment, such as light interference and ranging distance, which can affect measurement results. Passive depth acquisition utilizes feature points in the target scene image and predicts image depth information using computer vision algorithms. The entire process is simple to operate and requires no additional equipment, making it highly practical. However, the main source of information in the passive depth acquisition method is images from different perspectives. The captured images may have occluded areas, external light reflections, different light intensities and repeated patterns in the images, which will cause errors in the depth estimation results of the target scene.

[0004] Among the passive depth acquisition methods, the methods of using images to estimate depth are generally divided into two types: those based on traditional computer vision algorithm theory and those based on deep learning network framework.

[0005] The traditional multi-view depth estimation algorithm mainly consists of Structure from Motion (SFM) and Multi-view Stereo Matching (MVS):

[0006] 1. SFM algorithm: Estimates camera motion and depth information of the target scene from a set of 2D images from different perspectives. First, feature points are extracted from the images, and feature matching is performed between every two images. The spatial points corresponding to the matched feature points are then reconstructed using epipolar geometry. The resulting camera pose and spatial coordinates of the feature points are then optimized using the Bundler Adjustment method. Finally, the remaining images with the most matching points to the obtained spatial points are added, and Bundler Adjustment optimization is continued. Once all spatial points are obtained, a global Bundler Adjustment is performed. The SFM algorithm provides the camera position matrix and the initial sparse spatial points of the scene to the MVS algorithm.

[0007] 2. MVS algorithm: Based on the camera position parameters calculated by the SFM method, it performs a further dense reconstruction and outputs dense 3D spatial points. The MVS algorithm is implemented in multiple ways: point cloud, depth map, and voxel.

[0008] The features used in traditional depth estimation algorithms are manually designed, and it is difficult to extract features in areas with weak textures. As a result, the depth information in weak texture areas may be inaccurately predicted and a lot of manpower and time are required to design features in the early stages. Summary of the Invention

[0009] Therefore, the technical problem to be solved by the present invention is to provide a multi-view depth estimation method that can solve the existing problems of inaccurate or unpredictable depth information predicted by traditional depth estimation methods due to occlusion problems, weak texture areas, reflective surfaces, and repeated patterns in the captured images. In order to obtain depth information more accurately and quickly, a multi-view depth estimation method is provided to overcome the shortcomings of traditional depth estimation methods.

[0010] In order to solve the above problems, the present invention provides a multi-view depth estimation method, the specific steps are as follows:

[0011] Step 1: Image input: Obtain N+1 different images through the camera, and the front view image and the images in other directions are used as reference images I i＝0 and target image I i , the value range of i is 0 to N;

[0012] Step 2: Feature extraction: including FPN network module and CA module;

[0013] Step 2.1: The FPN network module extracts initial feature maps of different scales based on the image obtained in step 1;

[0014] Step 2.2: After the FPN network module extracts features at three different scales on the input camera image, the resulting initial feature map is transferred to the CA module through the DCN module.

[0015] Step 3: Depth Refinement: After feature extraction, feature maps of different scales are obtained and depth maps of different resolutions are predicted in a cascade manner.

[0016] Step 4: Depth Optimization: Using the residual learning network, the initial depth map with a resolution of W×H output by the depth refinement module is refined to obtain an optimized depth map. Focal loss is used to train the network model. The Adam optimization method is used to perform gradient updates on the network model based on the total loss, thereby guiding the training of the entire model.

[0017] Optionally, step 2.2 includes a CA module that embeds the attention information of the feature map along the horizontal and vertical directions into the channel. The specific steps are as follows:

[0018] Step 2.2.1: The CA module performs global average pooling on the input features in both horizontal and vertical directions, as shown in the following formulas (1) and (2):

[0019]

[0020]

[0021] Among them, the input tensor is X = [x1, x2, ..., x c ]∈R W×H×C , {W, H, C} represents the image width, height and number of channels; two pooling layers of W×1 and 1×H are used to encode the channels, (w, j) and (i, h) represent the input tensor x respectively c The image coordinate position, is the output result of the cth channel at the vertical direction h, is the output result of the cth channel at the horizontal direction w;

[0022] Step 2.2.2: The output of the pooling layer in the horizontal and vertical directions is P w and p h , then perform the Concate operation, as shown in formula (3):

[0023]

[0024] in, Represents the Concate operation, P is the output result after the Concate operation, P w is the output result of the pooling layer along the horizontal direction, p h Output result of the pooling layer along the vertical direction;

[0025] Step 2.2.3: Send the Concate output result to the 1×1 convolution layer, BN layer, and non-line activation function to obtain the intermediate feature map, as shown in the following formula (4):

[0026] f=δ(F 1×1 (P)) (4)

[0027] Among them, P is the output result of Concate operation, F 1×1 is a convolution transformation with a convolution kernel size of 1×1, δ is a nonlinear activation function, f is the intermediate feature map after encoding the spatial information of the input feature map along the horizontal and vertical directions, f∈R C / r×(H+W) , C is the number of channels, r is the channel reduction rate, W and H are the image width and height;

[0028] Step 2.2.4: Split the intermediate feature map f into two separate tensors f along the horizontal and vertical directions w ∈R C / r×W×H and f h ∈R C / r×W×H , respectively using 1×1 convolution operation, the two separate tensors are then processed by Sigmoid activation function to find the horizontal and vertical attention weights, respectively q w and q h , as shown in formula (5) and formula (6):

[0029] q w =σ(F 1×1 (f w )) (5)

[0030] q h =σ(F 1×1 (f h )) (6)

[0031] Among them, q w is the horizontal attention weight, q h is the vertical attention weight, σ is used to represent the Sigmoid activation function operation, and F 1×1 represents a 1×1 convolution operation, f w and f h represent the intermediate eigenvectors along the horizontal and vertical directions respectively;

[0032] Step 2.2.5: Horizontal Attention q w and vertical attention q h With input features X = [x1, x2, ..., x c ]∈RW×H×C Weighted summation is performed to obtain the final output tensor Y = [y1, y2, ..., y c ], as follows formula (7):

[0033]

[0034] where x c (i, j) is the feature tensor x at image coordinate (i, j) c , is the attention weight of the cth channel along the horizontal direction, is the attention weight of the cth channel along the vertical direction.

[0035] Optionally, step 3 includes cost volume construction, cost volume regularization, and depth estimation, specifically including the following steps:

[0036] Step 3.1: Construct the cost volume: Send the feature map output by the CA module to the cost volume construction module to construct the cost volume;

[0037] Step 3.2: Cost volume regularization: Regularize the cost volume using 3D convolution;

[0038] Step 3.3: Depth estimation: Use the Softmax operation to normalize the regularized cost volume to obtain a probability volume, and predict the depth map from the probability volume.

[0039] Optionally, step 3.1 further includes the following steps:

[0040] Step 3.1.1: Create a hypothetical depth plane for each pixel in the reference image based on the depth hypothesis range;

[0041] Step 3.1.2: Use homography to transform the 2D features of each target image to the hypothetical plane of the reference image.

[0042] A feature body is formed in the surface, and the homography transformation process is as follows:

[0043]

[0044] Among them, H i (d) represents the homography transformation matrix between the target image feature map and the reference feature map when the depth is d, i is the feature map number 0 to N, K i 、R i , t i are the camera intrinsic parameters, rotation matrix and translation vector when shooting the target image; and t0 are the inverse matrix operation of the camera intrinsic parameter matrix when shooting the reference image, the transpose matrix operation of the rotation matrix and the translation vector, I is the identity matrix, n0 Tis the main axis direction of the reference camera;

[0045] Step 3.1.3: Use the cost metric of variance to aggregate multiple feature volumes into a cost volume and calculate the cost volume as follows:

[0046]

[0047] Among them, V cost As the cost body, the N target image feature bodies V i and reference view features Perform variance operation.

[0048] Optionally, step 3.2 further includes the following steps:

[0049] Step 3.2.1: Downsample each feature layer through 3D convolution and maximum pooling layers;

[0050] Step 3.2.2: The obtained feature layer is then subjected to 3D deconvolution and upsampling operations.

[0051] Optionally, step 4 further includes the following steps:

[0052] Step 4.1: The reference image and the original scale feature depth estimation results of the depth refinement module are used as inputs to the residual learning network to obtain the optimized depth map;

[0053] Step 4.2: Use Focal loss as the loss function for model training. The cross entropy loss function is as shown in formula (10):

[0054]

[0055] Among them, L CE is the cross entropy loss function, is the predicted probability of pixel p at depth hypothesis d, is the depth value closest to the true value, p v is a subset of pixels with true values;

[0056] Step 4.3: The balanced cross entropy loss function is commonly used in the direction of target detection and classification, and its mathematical expression is as shown in formula (11):

[0057]

[0058] Among them, L BL In order to balance the cross entropy loss function, the weight value α is introduced on the basis of the cross entropy loss function, α∈[0,1]; is the predicted probability of pixel p at depth hypothesis d, is the depth value closest to the true value, p vis a subset of pixels with true values;

[0059] Step 4.4: Focal loss function is as shown in formula (12):

[0060]

[0061] Among them, L FL represents Focal loss, is the adjustment factor, γ is the parameter, is the predicted probability of pixel p at depth hypothesis d, is the depth value closest to the true value, p v is a subset of pixels with true values;

[0062] Step 4.5: Update the network model parameters using the Adam optimization method based on the total loss.

[0063] Beneficial effects

[0064] Embodiments of the present invention provide a multi-view depth estimation method that incorporates the end-to-end network model training concept from deep learning to obtain depth information. The entire algorithm framework can rapidly process multiple images to generate reliable depth information. Traditional depth estimation algorithms require hand-crafted features to calculate the cost volume, making feature extraction difficult in non-Lambertian scenes. However, the depth estimation method based on deep learning achieves accurate and robust depth estimation results even in weakly textured areas, reflective surfaces, repetitive patterns, and occluded regions. The method utilizes an FPN network to extract depth image features at three different scales from the input image. The initial features extracted at different scales by the FPN network are then transferred to a CA module via a DCN module. The CA module extracts position information from the initial feature map in the horizontal and vertical directions, then performs convolution operations on the features in both directions to obtain a global receptive field. The resulting feature map can be roughly viewed as a feature map with global contextual feature information. Since the FPN network, as a basic feature extraction module, extracts relatively local neighborhood contextual feature information, a DCN module is required between the FPN network and the CA module to adaptively adjust the feature extraction range using an additional offset. Some networks directly construct the cost volume after extracting features of different scales from the FPN module, ignoring global context information. The method of the present invention adds a CA module to enhance the expressiveness of the feature map by embedding position information into the channel, which has a beneficial effect on the accuracy of the output results of different functional modules in the entire network.

[0065] The method of the present invention uses the Focal Loss function to train the neural network framework. This is because in complex scene data sets, the Focal Loss function can process the image boundary area with higher accuracy than the results obtained using the cross entropy loss function.

[0066] Currently, due to the continuous improvement of hardware performance, the technology of applying deep learning to depth estimation algorithms has been rapidly developing. The multi-view depth estimation method based on deep learning specifically implements the algorithm by using a neural network framework to complete the entire process from 2D images to predicted depth maps.

[0067] Multi-view depth estimation method based on deep learning: By building a network framework for estimating image depth, after inputting a group of images, the corresponding depth map of each image can be predicted. The difference between the predicted depth map and the true depth image is calculated through the loss function. The Adam method is then used to continuously train and optimize the network parameters. Finally, the network with the smallest loss value is used as the trained model and directly used to predict the depth map. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is a general network framework diagram of an embodiment of the present invention;

[0069] Figure 2 This is a CBR module diagram in feature extraction according to an embodiment of the present invention;

[0070] Figure 3 This is a diagram of the out module in feature extraction in an embodiment of the present invention;

[0071] Figure 4 This is a DCN module diagram for feature extraction in an embodiment of the present invention;

[0072] Figure 5 This is a diagram of a CA module in feature extraction according to an embodiment of the present invention;

[0073] Figure 6 3D UNet module diagram in the regularized cost volume of an embodiment of the present invention. DETAILED DESCRIPTION

[0074] See also Figures 1 to 6 As shown, according to an embodiment of the present invention, a multi-view depth estimation method comprises the following specific steps:

[0075] Step 1: Image input: Obtain N+1 different images through the camera, and the front view image and the images in other directions are used as reference images I i＝0 and target image I i , the value range of i is 0 to N;

[0076] Furthermore, a set of multi-view images of the target scene is input, that is, N+1 views from different angles are taken by the camera, one of which is the reference image and the remaining N images are all used as target images. The reference image and its target image are sent to the feature extraction part of the network framework.

[0077] Step 2: Feature extraction: including FPN network module and CA module;

[0078] Step 2.1: The FPN network module extracts initial feature maps of different scales based on the image obtained in step 1.

[0079] Furthermore, N+1 images are first input into the feature pyramid network (FPN) for feature extraction. Let the input image be I i , the value range of i is 0 to N, I i＝0 is the reference image, N is the number of target images, width and height are denoted by W and H respectively, and the resolution is W×H. After N+1 images pass through the FPN network, each image will obtain initial feature maps with resolutions of W / 4×H / 4, W / 2×H / 2, and W×H. Using features at different scales can expand the feature receptive field. The output initial feature maps of different scales are input to the CA (Coordinate Attention, CA) module.

[0080] Furthermore, we first input the image I with a resolution of W×H i , use the FPN network that can output three feature scales for preliminary feature map extraction. The FPN network consists of three CBR modules and three out output feature modules: Among them, the CBR module is as follows Figure 2 As shown in Figure 1, it consists of a convolution layer, a BN (Batch Normalize) layer, and a ReLu activation function. i First, it passes through the CBR0 module, which consists of two identical convolutional layers, a BN layer and a ReLu activation function. The convolution kernel size of the convolution layer is 3×3, the number of output channels is 8, and the stride is 1; the output result of CBR0 is then input into the CBR1 module, which consists of three convolutional layers, a BN layer and a ReLu activation function. The convolution kernel sizes of the three convolutional layers are 5×5, 3×3, and 3×3 respectively, the output channels are all 16, and the strides are 2, 1, and 1 respectively; the output result of CBR1 is then input into the CBR2 module. Compared with the CBR1 module, the CBR2 module is the same as the CBR1 module except that the number of channels is 32.

[0081] Furthermore, the Batch Normalization (BN) layer is a batch normalization method, typically used before the activation function. It regularizes the features of a layer, calculating the mean and variance to make the model more stable, accelerating training speed and convergence. The BN layer generally performs a linear normalization on each pixel in the feature map after the convolutional layer operation.

[0082] The ReLu activation function not only accelerates network convergence but also alleviates the vanishing gradient problem. It is a new activation function proposed to improve the saturation of the Sigmoid function. The Sigmoid function causes the output of each neuron to range from 0 to 1, making it suitable for models that use predicted probabilities as output. However, the Sigmoid function itself has a disadvantage: saturation can cause the gradient to vanish.

[0083] Furthermore, the out module is used to obtain feature maps of different scales, such as Figure 3 As shown in the figure, first, the output of CBR2 is used as the input of the out0 module. Out0 performs a 1×1 convolution operation on the output of CBR2 and outputs a feature map of scale W / 4×H / 4. Next, the output of the out0 module and the output of the CBR1 module are used as the input of the out1 module. The out1 module upsamples the output of out0 by a factor of 2 and adds it to the output of the CBR1 module. The output of the added output is then subjected to a 1×1 convolution operation. The output of out1 is a feature map of scale W / 2×H / 2. Finally, the output of out1 and the output of CBR0 are used as the input of out2. The output of out1 is also upsampled by a factor of 2 and added to the output of CBR0. The output of the previous addition is then subjected to a 1×1 convolution operation. The output of the out2 module is a feature map of scale W×H. Through these steps, the FPN network outputs resolutions of W / 4×H / 4, W / 2×H / 2, and W×H, respectively. N+1 sets of multi-scale feature maps serve as the input of the CA module.

[0084] Step 2.2: After the FPN network module extracts features at three different scales on the input camera image, the obtained initial feature map is transitioned to the CA module through the DCN module.

[0085] Furthermore, the FPN network extracts initial feature maps of different scales and uses deformable convolution (DCN) to transfer these initial features to the subsequent CA module. The CA module performs feature operations on the initial feature maps of different scales to obtain the global receptive field. Finally, N+1 sets of feature maps of different scales are output, with sizes of W / 4×H / 4, W / 2×H / 2, and W×H, respectively, corresponding to 32, 16, and 8 channels. These N+1 sets of feature maps of different scales are then fed into the deep refinement process.

[0086] Furthermore, the CA module: After the FPN network extracts features at three different scales from the input camera image, the obtained initial feature map is transferred to the CA module through the DCN module. Figure 4 As shown in the figure, the DCN module consists of a deformable convolution layer (Deformable conv), a BN layer and a ReLu activation function layer, where the convolution kernel size of the deformable convolution layer is 3×3, the stride is 1, the padding is 1, and the deformable convolution group is 1.

[0087] The difference between deformable convolution and standard convolution is that, after the kernel size is determined, the sampling points of the standard convolution operation are arranged in a very regular square. Deformable convolution, however, adds an offset learned through an additional convolution layer to each sampling point, making the order of the sampling points irregular. Alternatively, the same offset can be added to each sampling point to achieve the effect of varying the scale of the sampling area. The deformable convolution module simultaneously inputs the feature map and the offset as inputs to the deformable convolution layer, which first offsets the sampling points before performing the convolution.

[0088] Furthermore, the CA module embeds the attention information of the feature map in the horizontal and vertical directions into the channel, which significantly improves the model performance. The CA module can be regarded as a computing unit, which takes the input tensor X = [x1, x2, ..., x c ]∈R W×H×C After processing by the CA module, the output is the transformation tensor Y=[y1,y2,…,y c ] CA module diagram Figure 5 As shown:

[0089] Step 2.2.1: The CA module performs global average pooling on the input features in both horizontal and vertical directions, as shown in the following formulas (1) and (2):

[0090]

[0091]

[0092] Among them, the input tensor is X = [x1, x2, ..., x c ]∈R W×H×C , {W, H, C} represents the image width, height and number of channels; two pooling layers of W×1 and 1×H are used to encode the channels, (w, j) and (i, h) represent the input tensor x respectively c The image coordinate position, is the output result of the cth channel at the vertical direction h, is the output result of the cth channel at the horizontal direction w;

[0093] Step 2.2.2: Next, perform concatenation on the pooling layer outputs in both horizontal and vertical directions. represents the Concate operation, and P is the output result after the Concate operation. This step is expressed by formula (3):

[0094]

[0095] in, Represents the Concate operation, P is the output result after the Concate operation, P w is the output result of the pooling layer along the horizontal direction, p h Output result of the pooling layer along the vertical direction.

[0096] Step 2.2.3: Concatenate the output results into a 1×1 convolution layer, a batch normalization layer, and a non-line activation function to obtain the intermediate feature map, which can be expressed as follows:

[0097] f=δ(F 1×1 (p)) (4)

[0098] Among them, P is the output result of Concate operation, F 1×1 is a convolution transformation with a convolution kernel size of 1×1, δ is a nonlinear activation function, f is the intermediate feature map after encoding the spatial information of the input feature map along the horizontal and vertical directions, f∈R C / r×(H+W) , C is the number of channels, r is the channel reduction rate, W and H are the image width and height;

[0099] Step 2.2.4: Split the intermediate feature map f into two separate tensors f along the horizontal and vertical directions w ∈R C / r×W×H and f h ∈R C / r×W×H , respectively using 1×1 convolution operation, the two separate tensors are then processed by Sigmoid activation function to find the horizontal and vertical attention weights, respectively q w and q h , as shown in formula (5) and formula (6):

[0100] q w =σ(F 1×1 (f w )) (5)

[0101] q h =σ(F 1×1 (f h )) (6)

[0102] Among them, qw is the horizontal attention weight, q h is the vertical attention weight, σ is used to represent the Sigmoid activation function operation, and F 1×1 represents a 1×1 convolution operation, f w and f h represent the intermediate eigenvectors along the horizontal and vertical directions respectively;

[0103] Step 2.2.5: Horizontal Attention q w and vertical attention q h With input features X = [x1, x2, ..., x c ]∈R W×H×C Weighted summation is performed to obtain the final output tensor Y = [y1, y2, ..., y c ], as follows formula (7):

[0104]

[0105] where x c (i, j) is the feature tensor x at image coordinate (i, j) c , is the attention weight of the cth channel along the horizontal direction, is the attention weight of the cth channel along the vertical direction.

[0106] As can be seen from the above, the CA module is different from the channel attention that converts the feature tensor into a single feature vector through two-dimensional global pooling. The CA module decomposes the channel attention into one-dimensional feature encodings along two spatial directions and then aggregates the features in these two directions. In this way, long-range dependencies can be obtained along one spatial direction, and precise position information can be retained along the other spatial direction. Through these two steps, the CA module uses precise position information to encode the channel relationship and long-range dependency into a pair of attention maps. The input feature map is then applied to the input tensor simultaneously in the horizontal and vertical directions. The attention map can accurately locate the exact position of the object of interest in the input feature map. Therefore, the CA module can better assist the entire model in recognition and obtain more useful accuracy information. Finally, the output of the CA module serves as the input of the deep refinement module.

[0107] Step 3: After feature extraction, feature maps of different scales are obtained, and depth maps of different resolutions are predicted in a cascade manner. The depth prediction result of the previous low-resolution feature map guides the depth range of the next high-resolution feature map when assuming parallel planes. The depth range of the depth prediction of the feature map with a resolution of W / 4×H / 4 is the entire depth range of the input scene. Due to the large depth range, the assumed plane interval is large, and a rough depth value is generated. In the subsequent depth estimation operations with resolutions of W / 2×H / 2 and W×H, a finer assumed plane interval is used to restore a more accurate depth estimate. The depth refinement module is divided into three parts: cost volume construction, cost volume regularization, and depth estimation.

[0108] Furthermore, in step 3.1: Cost Volume Construction, the feature maps output by the CA module are fed into the cost volume construction module. This cost volume construction module consists of three steps: First, a hypothetical depth plane is established for each pixel in the reference image based on the depth hypothesis range. Then, the 2D features of each target image are transformed into the hypothetical plane of the reference image using a homography to form a feature volume. Finally, a variance cost metric is used to aggregate multiple feature volumes into a cost volume.

[0109] Furthermore, constructing a 3D cost volume from the feature map output by the CA module requires obtaining the depth information range and camera parameters through differentiable homography changes. The method of the present invention adopts a cascaded cost volume construction method. This not only reduces GPU usage but also obtains more accurate depth information through depth prediction values at different scales.

[0110] The homography transform projects the target image feature map onto several parallel planes under the reference image. This process is similar to the plane scanning algorithm in 3D reconstruction. Under the reference image coordinates, the target image feature map is encoded using camera parameters and converted to the coordinate system corresponding to the reference image through the homography transform. Assuming that there are N target images in a set of input images, these N mapped feature maps constitute M feature bodies. The target image feature map is mapped to the reference image coordinate system. Different depths result in different sizes of the mapped feature maps. Since the feature volumes formed by the feature maps have different spatial sizes, a bilinear interpolation algorithm needs to be used for each feature map to make the mapped feature maps have the same height and width. The homography transform determines the coordinate change from the target image feature map to the cost volume at the depth value d. The transformation formula (8) is as follows:

[0111]

[0112] Among them H i (d) represents the homography transformation matrix between the target image feature map and the reference feature map when the depth is d, i is the feature map number 0 to N, K i 、R i , ti are the camera intrinsic parameters, rotation matrix and translation vector at other perspectives; and t0 are the camera intrinsic parameters, rotation matrix and translation vector under the reference perspective, I is the identity matrix, and n0 is the main axis direction of the reference camera.

[0113] The cost metric of variance is used to aggregate multiple feature volumes into a cost volume and calculate the cost volume as follows:

[0114]

[0115] Among them, V cost As the cost body, the N target image feature bodies V i and reference view features Perform variance operation;

[0116] Step 3.2: Cost Regularization: Use 3D convolution to regularize the cost volume. By constructing a smooth and dense matching relationship, reducing the impact of noise on the original cost volume, and aggregating context information to normalize the original cost volume, a regularized cost volume is obtained.

[0117] Furthermore, due to interference factors such as occlusion or weak texture areas, the generated cost volume contains noise, so it is necessary to regularize the cost volume to obtain a standardized cost volume. The initial cost volume is regularized using multi-scale 3D convolution, and the 3D convolution part is a 3D UNet structure. The regularized cost volume is then used to generate a probability volume for depth prediction through a Softmax operation.

[0118] like Figure 6 As shown, the 3D UNet network has an encoder-decoder structure: each feature layer is first downsampled through 3D convolution and max pooling layers, and then the resulting feature layer is upsampled through 3D deconvolution. During training, the encoder and decoder are connected using skip-layer connections to fuse the downsampled and upsampled outputs of the same scale, effectively preventing the vanishing gradient problem. Specifically, during the encoding process, features are input through 3D convolution, then gradually downsampled to 1 / 2, 1 / 4, and 1 / 8 of the original resolution through max pooling layers with a stride of 2. In the feature decoding part, the resulting features are upsampled to 1 / 4, 1 / 2, and the original feature resolution through 3D deconvolution and upsampling.

[0119] Step 3.3: Depth Estimation:

[0120] The regularized cost volume is normalized using the Softmax operation to obtain a probability volume, from which the depth map is predicted. A Soft Argmin operation is used to uniformly sample within the depth hypothesis range. The expected value here produces a continuous depth estimate, outputting the initial depth map. The depth information obtained also guides the depth range of the next-scale image.

[0121] Further, such as Figure 1 In the depth estimation part, to estimate the depth value of each point, the cost volume is transformed into a probability volume along the depth direction using the Softmax function. A Soft argmin operation is then used to uniformly sample within the depth hypothesis range, outputting an initial depth map. The Softmax operation compresses the depth dimension information of the regularized cost volume to a distribution between 0 and 1. The resulting probability volume can not only infer pixel depth but also measure the confidence of the estimate. After the Soft argmin operation obtains the initial depth map for each image, the depth range at the current scale is adaptively sampled based on the depth map of the previous scale, obtaining a smaller depth interval for more accurate values in subsequent depth estimation.

[0122] Step 4: Depth optimization: Using the residual learning network, the W×H initial depth map output by the depth refinement module is refined to obtain a reference depth map. Focal loss is used to train the network model. The Adam optimization method is used to perform gradient updates on the network model based on the total loss, thereby guiding the training of the entire model.

[0123] Step 4.1: If Figure 1 The depth optimization module uses the reference image and the original-scale feature depth estimation results from the depth refinement module as input to the residual learning network to obtain the optimized depth map. Specifically, the original-resolution depth map predicted from the probability volume is refined using the reference image as a guide. Four convolution operations with a kernel size of 3×3 and a stride of 1 are performed, and the output is an optimized reference depth map with 1 channel.

[0124] During network training, the loss function is used to assess the difference between the model's predictions and the true values. The error between the predictions and the true values after each forward iteration of the network is calculated, guiding the next step in deep learning model training.

[0125] The loss function for training networks has traditionally been based on depth regression, where the absolute value between the predicted value and the true value is used as the loss. The proposed method instead treats depth estimation as a classification task to strengthen training supervision in complex areas, using focal loss to train the network model.

[0126] The mean squared error (MSE) loss function is a common regression-based loss function. The MSE loss function is calculated by squaring the difference between the model's predicted value and the true value to find the expected value. A smaller expected value indicates higher model prediction accuracy. While the MSE loss function is straightforward and simple to use, it can be slow to learn when using gradient descent with the Softmax function to determine probability.

[0127] Step 4.2: The cross entropy loss function is used to calculate the probability of different categories. It is easy to perform derivative calculations using the gradient descent method, and the model learning speed is fast. CE is the cross entropy loss function, as shown in formula (10):

[0128]

[0129] Among them, L CE is the cross entropy loss function, is the predicted probability of pixel p at depth hypothesis d, is the depth value closest to the true value, p v is the subset of pixels with true values.

[0130] Focal loss belongs to the classification loss function. The commonly used classification loss function is the cross entropy loss function.

[0131] Step 4.3: The cross entropy loss function is commonly used in the direction of target detection and classification. In order to balance the common problem of imbalanced foreground and background classification results, the weight α∈[0,1] is introduced on the basis of the cross entropy loss function. The balanced cross entropy loss function is as shown in formula (11):

[0132]

[0133] Among them, L BL In order to balance the cross entropy loss function, a weight value α is introduced based on the cross entropy loss function to balance the common problem of imbalanced foreground and background classification results, α∈[0,1]; is the predicted probability of pixel p at depth hypothesis d, is the depth value closest to the true value, p v is the subset of pixels with true values.

[0134] Although formula (11) can solve the problem of classification imbalance, it cannot make the model pay more attention to the difficult-to-classify areas. Therefore, in step 4.4: Focal loss function, in order to make the network pay more attention to the difficult-to-process areas, the cross entropy loss function structure is readjusted and the cross entropy loss function is added. factor, γ is a parameter. Focalloss uses LFL Expressed as formula (12):

[0135]

[0136] Among them, L FL represents Focal loss, is the adjustment factor, γ is the parameter, is the predicted probability of pixel p at depth hypothesis d, is the depth value closest to the true value, p v is the subset of pixels with true values.

[0137] The Focal loss function can adjust the loss weights for different regions of complexity: when γ = 0, the Focal loss function uses the cross-entropy loss function; when γ = 2, the Focal loss function is suitable for complex scenes, making Focal loss more accurate than the cross-entropy loss function. In complex scene datasets such as Tanks and Temples, the Focal loss function is more suitable for the diverse and complex scenes in the dataset. In particular, the processing accuracy of image boundary areas is higher than that obtained with the cross-entropy loss function, enabling the trained model to obtain the best depth estimation value.

[0138] Step 4.5: Finally, use the Adam optimization method to backpropagate and update the network model parameters according to the total loss.

[0139] Currently, due to the continuous improvement of hardware performance, the technology of applying deep learning to depth estimation algorithms has been rapidly developing. The multi-view depth estimation method based on deep learning specifically implements the algorithm by using a neural network framework to complete the entire process from 2D images to predicted depth maps.

[0140] Multi-view depth estimation method based on deep learning: By building a network framework for estimating image depth, after inputting a group of images, the corresponding depth map of each image can be predicted. The difference between the predicted depth map and the true depth image is calculated through the loss function. The Adam method is then used to continuously train and optimize the network parameters. Finally, the network with the smallest loss value is used as the trained model and directly used to predict the depth map.

[0141] It is easy for those skilled in the art to understand that, under the premise of no conflict, the above-mentioned advantageous methods can be freely combined and superimposed.

Claims

1. A multi-view depth estimation method, characterized in that: The specific steps are as follows: Step 1: Image input: Obtain N+1 different images through the camera, and the front view image and the images in other directions are used as reference images I i＝0 and target image I i , the value range of i is 0 to N; Step 2: Feature extraction: including FPN network module and CA module; Step 2.1: The FPN network module extracts initial feature maps of different scales based on the image obtained in step 1; Step 2.2: After the FPN network module extracts features at three different scales on the input camera image, the resulting initial feature map is transferred to the CA module through the DCN module. Step 3: Depth Refinement: After feature extraction, feature maps of different scales are obtained and depth maps of different resolutions are predicted in a cascade manner. Step 4: Depth Optimization: Using the residual learning network, the initial depth map with a resolution of W×H output by the depth refinement module is refined to obtain an optimized depth map. Focalloss is used to train the network model. The Adam optimization method is used to perform gradient updates on the network model based on the total loss, thereby guiding the training of the entire model. Among them, step 2.2 includes the CA module, which embeds the attention information of the feature map along the horizontal and vertical directions into the channel. The specific steps are as follows: Step 2.2.1: The CA module performs global average pooling on the input features in both horizontal and vertical directions. Step 2.2.2: The output of the pooling layer in the horizontal and vertical directions is P w and p h , then perform the Concate operation; Among them, step 3 includes three parts: cost volume construction, cost volume regularization and depth estimation, which specifically include the following steps: Step 3.1: Construct the cost volume: Send the feature map output by the CA module to the cost volume construction module to construct the cost volume; Step 3.2: Cost volume regularization: Regularize the cost volume using 3D convolution; Step 3.3: Depth estimation: Use the Softmax operation to normalize the regularized cost volume to obtain a probability volume, and predict the depth map from the probability volume.

2. The multi-view depth estimation method according to claim 1, wherein: Step 2.2 includes the CA module, which embeds the attention information of the feature map along the horizontal and vertical directions into the channel. The specific steps are as follows: Step 2.2.1: The CA module performs global average pooling on the input features in both horizontal and vertical directions, as shown in the following formulas (1) and (2): Among them, the input tensor is X = [x1, x2, ..., x c ]∈R W×H×C , {W, H, C} represents the image width, height and number of channels; two pooling layers of W×1 and 1×H are used to encode the channels, (w, j) and (i, h) represent the input tensor x respectively c The image coordinate position, is the output result of the cth channel at the vertical direction h, is the output result of the cth channel at the horizontal direction w; Step 2.2.2: The output of the pooling layer in the horizontal and vertical directions is P w and p h , then perform the Concate operation, as shown in formula (3): in, Represents the Concate operation, P is the output result after the Concate operation, P w is the output result of the pooling layer along the horizontal direction, p h Output result of the pooling layer along the vertical direction; Step 2.2.3: Send the Concate output result to the 1×1 convolution layer, BN layer, and non-line activation function to obtain the intermediate feature map, as shown in the following formula (4): f=δ(F 1×1 (P)) (4) Among them, P is the output result of Concate operation, F 1×1 is a convolution transformation with a convolution kernel size of 1×1, δ is a nonlinear activation function, f is the intermediate feature map after encoding the spatial information of the input feature map along the horizontal and vertical directions, f∈R C / r×(H+W) , C is the number of channels, r is the channel reduction rate, W and H are the image width and height; Step 2.2.4: Split the intermediate feature map f into two separate tensors f along the horizontal and vertical directions w ∈R C / r×W×H and f h ∈R C / r×W×H , respectively using 1×1 convolution operation, the two separate tensors are then processed by Sigmoid activation function to find the horizontal and vertical attention weights, respectively q w and q h , as shown in formula (5) and formula (6): what w =σ(F 1×1 (f w )) (5) what h =σ(F 1×1 (f h )) (6) Among them, q w is the horizontal attention weight, q h is the vertical attention weight, σ is used to represent the Sigmoid activation function operation, and F 1×1 represents a 1×1 convolution operation, f w and f h represent the intermediate eigenvectors along the horizontal direction and the intermediate eigenvectors along the vertical direction respectively; Step 2.2.5: Horizontal Attention q w and vertical attention q h With input features X = [x1, x2, ..., x c ]∈R W ×H×C Weighted summation is performed to obtain the final output tensor Y = [y1, y2, ..., y c ], as follows formula (7): where x c (i, j) is the feature tensor x at image coordinate (i, j) c , is the attention weight of the cth channel along the horizontal direction, is the attention weight of the cth channel along the vertical direction.

3. The multi-view depth estimation method according to claim 1, wherein: Step 3.1 also includes the following steps: Step 3.1.1: Create a hypothetical depth plane for each pixel in the reference image based on the depth hypothesis range; Step 3.1.2: Use homography to transform the two-dimensional features of each target image into the hypothetical plane of the reference image to form a feature volume. The homography transformation process is as follows: Among them, H i (d) represents the homography transformation matrix between the target image feature map and the reference feature map when the depth is d, i is the feature map number 0 to N, K i 、R i , t i are the camera intrinsic parameters, rotation matrix and translation vector when shooting the target image; and t0 are the inverse matrix operation of the camera intrinsic parameter matrix when shooting the reference image, the transpose matrix operation of the rotation matrix and the translation vector, I is the identity matrix, n0 T is the main axis direction of the reference camera; Step 3.1.3: Use the cost metric of variance to aggregate multiple feature volumes into a cost volume and calculate the cost volume as follows: Among them, V cost As the cost body, the N target image feature bodies V i and reference view features Perform variance operation.

4. The multi-view depth estimation method according to claim 1, wherein: Step 3.2 also includes the following steps: Step 3.2.1: Downsample each feature layer through 3D convolution and maximum pooling layers; Step 3.2.2: The obtained feature layer is then subjected to 3D deconvolution and upsampling operations.

5. The multi-view depth estimation method according to claim 1, wherein: Step 4 also includes the following steps: Step 4.1: The reference image and the original scale feature depth estimation results of the depth refinement module are used as inputs to the residual learning network to obtain the optimized depth map; Step 4.2: Use Focal loss as the loss function for model training. The cross entropy loss function is as shown in formula (10): Among them, L CE is the cross entropy loss function, is the predicted probability of pixel p at depth hypothesis d, is the depth value closest to the true value, p v is a subset of pixels with true values; Step 4.3: The balanced cross entropy loss function is commonly used in the direction of target detection and classification, and its mathematical expression is as shown in formula (11): Among them, L BL In order to balance the cross entropy loss function, the weight value α is introduced on the basis of the cross entropy loss function, α∈[0,1]; is the predicted probability of pixel p at depth hypothesis d, is the depth value closest to the true value, p v is a subset of pixels with true values; Step 4.4: Focal loss function is as shown in formula (12): Among them, L FL represents Focal loss, is the adjustment factor, γ is the parameter, is the predicted probability of pixel p at depth hypothesis d, is the depth value closest to the true value, p v is a subset of pixels with true values; Step 4.5: Update the network model parameters using the Adam optimization method based on the total loss.

Images