Image depth estimation method and device based on global and local features

Abstract

The application discloses a kind of based on global and local feature image depth estimation method and device, comprising: 1) data preprocessing;2) extract global feature and local feature in input image;3) fusion global feature and local feature;4) decoding prediction.The method of the present application introduces a new data enhancement method only for depth estimation task, which can enable the model to extract global features of color images in the encoding stage, and also extract local features, and in the decoding stage, using the fusion module designed by us, the dense, high-quality depth map is predicted while combining different features, and shows stronger generalization performance on additional data sets.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to an image depth estimation method and apparatus based on global and local features. Background technology:

[0002] Monocular depth estimation aims to map a single RGB image into a dense depth map. This is a classic task in computer vision, crucial for many practical applications such as autonomous driving, robot tracking and localization, and 3D scene reconstruction. However, this task has long suffered from an unforeseen problem: targets in a single color image may originate from multiple different 3D scenes. Therefore, how to improve model learning with limited available data has become a major research focus for researchers in this field.

[0003] Methods for predicting depth maps based on a single color image can be broadly categorized into three types: The first is the traditional machine learning-based approach, specifically constructing probabilistic models such as Markov Random Fields (MRFs) or Conditional Random Fields (CRFs) to describe the relationship between the depth of a pixel or region and the depths of its neighboring pixels or regions, thereby matching each pixel with the depth value that best matches the actual scene. This method has the following drawbacks: besides being overly complex in model construction and understanding and difficult to reproduce, the randomness of the model makes it impossible to accurately find depths in different scenes, affecting the final prediction results. The second method is based on supervised deep learning, which can be further divided into the following types: 1) using only convolution to extract encoded information from the image to predict depth; this method lacks global information in the image. 2) using only self-attention mechanisms to extract encoded information from the image to predict depth; this method lacks local information. 3) using convolution and self-attention mechanisms separately in the encoding and decoding stages of the model; this method treats the two types of information separately and does not fully combine their characteristics. The third method is based on unsupervised deep learning, which generally uses the parallax principle to calculate the target depth. However, this method has the drawback that it requires the pre-obtaining of the left and right images as well as camera parameters, and cannot be applied to scenes under different devices, making it very inconvenient for practical engineering. Summary of the Invention

[0004] Objective: To address the shortcomings of existing methods in not fully utilizing global and local information, this invention provides an image depth estimation method and apparatus based on global and local features.

[0005] Technical solution: To solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0006] Firstly, an image depth estimation method based on global and local features is provided, including:

[0007] Obtain the color image to be estimated;

[0008] The color image is preprocessed to obtain the input image InputRGB;

[0009] Extracting global and local features from the input image InputRGB: The SwinTransformer-T module is used to extract the global features of the input image InputRGB, and the ConvNeXt-S module is used to extract the local features of the input image InputRGB.

[0010] The extracted global features and local features are fused to obtain a fused feature vector;

[0011] The fused feature vector is decoded and restored to the InputRGB size of the input image to obtain the absolute depth map Depth_pre. The absolute depth map Depth_pre is then used for mirror prediction to obtain the predicted depth map Depth_final, which is used as the image depth estimation result.

[0012] In some embodiments, the data preprocessing includes: performing data preprocessing using the CutDepth-Mask data augmentation method; it also includes: random horizontal flipping, random cropping, random brightness contrast, random gama regularization, and random saturation.

[0013] In some embodiments, the global features of the input image InputRGB are extracted using the SwingTransformer-T module, including:

[0014] The input image InputRGB is used as input to the SwinTransformer-T module. A convolutional layer flattens the four-dimensional input variable B x 3 x H x W into a three-dimensional variable B x P x C. The number of convolutional kernels is C x 3, the kernel size is 7, the stride is 4, and the padding is 2. The process is as follows:

[0015]

[0016] In the formula, The feature vector generated by the i-th convolutional layer after input image I; The weight parameters of the i-th convolutional kernel; The bias parameter of the i-th convolution kernel; *: convolution operation; the maximum value of i is C;

[0017] The three-dimensional variable I embThe data is fed into a global feature extraction module consisting of four self-attention layers and three downsampling layers for encoding. Each self-attention layer comprises multiple linear layers and different activation functions, while the downsampling layer consists of a convolutional layer of size 3 and a stride of 2. The process of the self-attention layer is illustrated by the following formula:

[0018] Q = (I emb ·W q +b q )

[0019] K = (I emb ·W k +b k )

[0020] V = (I emb ·W v +b v )

[0021]

[0022]

[0023] In the formula, Q, K, and V are three linear functions; W q W k W v These represent the weight parameters of the three linear layers q, k, and v, respectively; b q b k b v q, k, and v represent the bias parameters of the three linear layers, respectively; softmax represents the logistic regression function, as shown in the formula below:

[0024]

[0025] d: Dimension of feature vector K; G i : The global feature vector output by the i-th self-attention layer; The weight parameters of the first linear layer of the i-th self-attention layer; The bias parameters of the first linear layer of the i-th self-attention layer; The weight parameters of the second linear layer in the i-th self-attention layer; The bias parameters of the second linear layer in the i-th self-attention layer; · represents matrix multiplication, and T represents the transpose matrix;

[0026] The SwinTransformer-T module outputs four global feature vectors G1, G2, G3, and G4 at different scales, representing the input image's RGB dimensions respectively.

[0027] In some embodiments, the ConvNeXt-S module is used to extract local features of the input image InputRGB, including:

[0028] The input image InputRGB is used as input to the ConvNeXt-S module. The module encodes InputRGB using a four-layer convolutional block and a four-layer downsampling layer. Each convolutional block consists of one 7x7 kernel and two 1x1 kernels concatenated, outputting four local feature vectors L1, L2, L3, and L4 at different scales, representing the size of the input image InputRGB.

[0029] In some embodiments, the extracted global features are fused with local features, including:

[0030] The local features L1, L2, L3, L4 and the global features G1, G2, G3, G4 are input into the fusion module and fused to obtain the fused feature vectors M1, M2, M3, M4:

[0031]

[0032]

[0033]

[0034] In the formula: concat: concatenation; dim: concatenation dimension; Maxpool: max pooling, size 3, stride 1, padding 1; Avgpool: mean pooling, size 3, stride 1, padding 1; Conv: convolution, size 3, stride 1, padding 1; i: the i-th feature vector. These are intermediate parameters.

[0035] In some embodiments, decoding the fused feature vectors back to the InputRGB size to obtain the absolute depth map Depth_pre includes:

[0036] The fused feature vectors M1, M2, M3, and M4 are merged to obtain the merged fused feature vectors Y1, Y2, and Y3. Two consecutive convolutions are used to connect adjacent fused feature vectors M1, M2, and M3. i M i+1 :

[0037] Y1=Conv(Conv(concat(bilinear(M1),M2,dim=1)))

[0038] Y2=Conv(Conv(concat(bilinear(Y1),M3,dim=1)))

[0039] Y3=Conv(Conv(concat(bilinear(Y2),M4,dim=1)))

[0040] In the formula, bilinear: bilinear interpolation, scaling factor 2; dim: concatenation dimension; concat: concatenation; Conv: convolution, Y... i : Feature map after each scaling and merging;

[0041] Y3 is scaled to half the resolution of the input image (InputRGB) using the Bilinear scaling function. A convolutional network is used to enhance the network's fitting ability, and finally, Sigmoid normalization is applied to obtain the normalized Depth_pre_norm. The process is shown in the following formula:

[0042] Depth_pre_norm=Sigmoid(Conv(bilinear(Y3)))

[0043] The normalized Depth_pre_norm is multiplied by the preset maximum depth value max_depth, and then bilinear interpolation is used to restore it to the size of the input image InputRGB, thus obtaining the absolute depth map Depth_pre.

[0044] The absolute depth map Depth_pre includes the left image Depth_pre_L and the right image Depth_pre_R; the input image InputRGB is processed by the entire network model and outputs the left image Depth_pre_L and the right image Depth_pre_R of the absolute depth map, which can be expressed by a general model formula as follows:

[0045] Depth_pre_L = Net(InputRGB)

[0046] Depth_pre_R=Net(ship(InputRGB))

[0047] In the formula, Net: the entire network model; ship: horizontal flip.

[0048] In some embodiments, the absolute depth map Depth_pre is used to obtain the predicted depth map Depth_final by mirror prediction, including: the absolute depth map Depth_pre includes a left image Depth_pre_L and a right image Depth_pre_R of the absolute depth map;

[0049]

[0050] In the formula, "ship" means horizontal flipping.

[0051] In this embodiment, the extraction of global and local features from a color image can be performed using the following formula:

[0052] L = ConNeXt(InputRGB)

[0053] G = Swing(InputRGB)

[0054] Wherein, ConNeXt: local encoder; Swing: global encoder;

[0055] In addition, to better reflect the superior performance of our model, we also designed a new index function: Dh rel (Double hook function). The formula for the double hook function is shown below:

[0056]

[0057] In the formula, p i : Predict the effective depth of the depth map Depth_pre; g i : The effective depth of the true depth map Depth_gt.

[0058] In a second aspect, the present invention provides an image depth estimation device based on global and local features, including a processor and a storage medium;

[0059] The storage medium is used to store instructions;

[0060] The processor is configured to operate according to the instructions to perform the steps of the method according to the first aspect.

[0061] Thirdly, the present invention provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0062] The advantages of this invention are: the method provided by this invention enables the model to extract not only local information of the image but also global information during the encoding stage, with only a single color image as input. The decoding stage fully combines the two types of extracted information, thereby estimating the depth more accurately. Attached Figure Description

[0063] Figure 1 This is a flowchart of the method in an embodiment of the present invention.

[0064] Figure 2 This is a schematic diagram of the data augmentation method in an embodiment of the present invention.

[0065] Figure 3This is a structural diagram of the encoding stage in an embodiment of the present invention.

[0066] Figure 4 This is a structural diagram of the fusion module in an embodiment of the present invention.

[0067] Figure 5 This is a visualization of the method of this invention on the outdoor dataset KITTI. Detailed Implementation

[0068] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0069] In the description of this invention, "several" means one or more, "multiple" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0070] In the description of this invention, the terms "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0071] Example 1

[0072] An image depth estimation method based on global and local features includes:

[0073] Obtain the color image to be estimated;

[0074] The color image is preprocessed to obtain the input image InputRGB;

[0075] Extracting global and local features from the input image InputRGB: The SwinTransformer-T module is used to extract the global features of the input image InputRGB, and the ConvNeXt-S module is used to extract the local features of the input image InputRGB.

[0076] The extracted global features and local features are fused to obtain a fused feature vector;

[0077] The fused feature vector is decoded and restored to the InputRGB size of the input image to obtain the absolute depth map Depth_pre. The absolute depth map Depth_pre is then used for mirror prediction to obtain the predicted depth map Depth_final, which is used as the image depth estimation result.

[0078] In some embodiments, the data preprocessing includes: performing data preprocessing using the CutDepth-Mask data augmentation method;

[0079] It also includes: random horizontal flip, random cropping, random brightness contrast, random gama regular distribution, and random saturation.

[0080] Further, data preprocessing: Applying preprocessing methods to the given color image RGB and the true depth map Depth_gt, and using the preprocessed image as input to the model for subsequent depth map prediction is a common technique in deep learning. Common preprocessing methods include: random horizontal flipping, random cropping, random brightness contrast, random gamma regularization, and random saturation. We added a masked CutDepth method. The original CutDepth data augmentation method replaces randomly cropped regions in the color image with the same depth values, using the preprocessed InputRGB as the model input. However, because the obtained true depth labels are incomplete, i.e., the label depth map contains a large number of missing depth values. Missing values ​​are 0, but we know that 0 does not contribute to the network's learning in deep learning. Therefore, we believe that the original CutDepth method has problems in practical operation. CutDepth with a mask means that in order to eliminate missing values, we added a mask to the original CutDepth method, that is, to retain pixels with missing depth values ​​in the original region. Compared with the original CutDepth method, it can not only reduce computational complexity, but also improve the richness of model input.

[0081] In some embodiments, the global features of the input image InputRGB are extracted using the SwingTransformer-T module, including:

[0082] The input image InputRGB is used as input to the SwinTransformer-T module. A convolutional layer flattens the four-dimensional input variable B x 3 x H x W into a three-dimensional variable B x P x C. The number of convolutional kernels is C x 3, the kernel size is 7, the stride is 4, and the padding is 2. The process is as follows:

[0083]

[0084] In the formula, The feature vector generated by the i-th convolutional layer after input image I; The weight parameters of the i-th convolutional kernel; The bias parameter of the i-th convolution kernel; *: convolution operation; the maximum value of i is C;

[0085] The three-dimensional variable I emb The data is fed into a global feature extraction module consisting of four self-attention layers and three downsampling layers for encoding. Each self-attention layer comprises multiple linear layers and different activation functions, while the downsampling layer consists of a convolutional layer of size 3 and a stride of 2. The process of the self-attention layer is illustrated by the following formula:

[0086] Q = (I emb ·W q +b q )

[0087] K = (I emb ·W k +b k )

[0088] V = (I emb ·W v +b v )

[0089]

[0090]

[0091] In the formula, Q, K, and V are three linear functions; W q W k W v These represent the weight parameters of the three linear layers q, k, and v, respectively; b q b k b v q, k, and v represent the bias parameters of the three linear layers, respectively; softmax represents the logistic regression function, as shown in the formula below:

[0092]

[0093] d: Dimension of feature vector K; G i : The global feature vector output by the i-th self-attention layer; The weight parameters of the first linear layer of the i-th self-attention layer; The bias parameters of the first linear layer of the i-th self-attention layer; The weight parameters of the second linear layer in the i-th self-attention layer; The bias parameters of the second linear layer in the i-th self-attention layer; · represents matrix multiplication, and T represents the transpose matrix;

[0094] The SwinTransformer-T module outputs four global feature vectors G1, G2, G3, and G4 at different scales, representing the input image's RGB dimensions respectively.

[0095] In some embodiments, the ConvNeXt-S module is used to extract local features of the input image InputRGB, including:

[0096] The input image InputRGB is used as input to the ConvNeXt-S module. The module encodes InputRGB using a four-layer convolutional block and a four-layer downsampling layer. Each convolutional block consists of one 7x7 kernel and two 1x1 kernels concatenated, outputting four local feature vectors L1, L2, L3, and L4 at different scales, representing the size of the input image InputRGB.

[0097] In some embodiments, the extracted global features are fused with local features, including:

[0098] The local features L1, L2, L3, L4 and the global features G1, G2, G3, G4 are input into the fusion module and fused to obtain the fused feature vectors M1, M2, M3, M4:

[0099]

[0100]

[0101]

[0102] In the formula: concat: concatenation; dim: concatenation dimension; Maxpool: max pooling, size 3, stride 1, padding 1; Avgpool: mean pooling, size 3, stride 1, padding 1; Conv: convolution, size 3, stride 1, padding 1; i: the i-th feature vector. These are intermediate parameters.

[0103] In some embodiments, decoding the fused feature vectors back to the InputRGB size to obtain the absolute depth map Depth_pre includes:

[0104] The fused feature vectors M1, M2, M3, and M4 are merged to obtain the merged fused feature vectors Y1, Y2, and Y3. Two consecutive convolutions are used to connect adjacent fused feature vectors M1, M2, and M3. i M i+1 :

[0105] Y1=Conv(Conv(concat(bilinear(M1),M2,dim=1)))

[0106] Y2=Conv(Conv(concat(bilinear(Y1),M3,dim=1)))

[0107] Y3=Conv(Conv(concat(bilinear(Y2),M4,dim=1)))

[0108] In the formula, bilinear: bilinear interpolation, scaling factor 2; dim: concatenation dimension; concat: concatenation; Conv: convolution, Y... i : Feature map after each scaling and merging;

[0109] Y3 is scaled to half the resolution of the input image (InputRGB) using the Bilinear scaling function. A convolutional network is used to enhance the network's fitting ability, and finally, Sigmoid normalization is applied to obtain the normalized Depth_pre_norm. The process is shown in the following formula:

[0110] Depth_pre_norm=Sigmoid(Conv(bilinear(Y3)))

[0111] The normalized Depth_pre_norm is multiplied by the preset maximum depth value max_depth, and then bilinear interpolation is used to restore it to the size of the input image InputRGB, thus obtaining the absolute depth map Depth_pre.

[0112] The absolute depth map Depth_pre includes the left image Depth_pre_L and the right image Depth_pre_R; the input image InputRGB is processed by the entire network model and outputs the left image Depth_pre_L and the right image Depth_pre_R of the absolute depth map, which can be expressed by a general model formula as follows:

[0113] Depth_pre_L = Net(InputRGB)

[0114] Depth_pre_R=Net(ship(InputRGB))

[0115] In the formula, Net: the entire network model; ship: horizontal flip.

[0116] In some embodiments, the absolute depth map Depth_pre is used to obtain the predicted depth map Depth_final by mirror prediction, including: the absolute depth map Depth_pre includes a left image Depth_pre_L and a right image Depth_pre_R of the absolute depth map;

[0117]

[0118] In the formula, "ship" means horizontal flipping.

[0119] In some embodiments, the extraction of global and local features from a color image can be performed according to the following formula:

[0120] L = ConNeXt(InputRGB)

[0121] G = Swing(InputRGB)

[0122] Wherein, ConNeXt is the local encoder; Swing is the global encoder.

[0123] In addition, to better reflect the superior performance of our model, we also designed a new index function: Dh rel (Double hook function). The formula for the double hook function is shown below:

[0124]

[0125] In the formula, p i : Predict the effective depth of the depth map Depth_pre; g i : The effective depth of the true depth map Depth_gt.

[0126] In some embodiments, such as Figure 1 As shown, the image depth estimation method based on global and local features in this embodiment includes the following specific steps:

[0127] 1) First, a color image is preprocessed using the CutDepth-Mask data augmentation method to obtain a new input image, InputRGB.

[0128] 2) InputRGB is fed into the model's encoding stage to extract global features G1, G2, G3, G4 and local features L1, L2, L3, L4. The specific process is as follows:

[0129] 2.1) First, IuputRGB is used as the input of the global feature encoder SwinTransformer-T. SwinTransformer-T is divided into 4 layers. Each layer consists of a sliding window self-attention mechanism and outputs feature maps of multiple scales. The module names and output feature map sizes are g1: 88x176, g2: 44x88, g3: 22x44, and g4: 11x22, respectively.

[0130] 2.2) IuputRGB is still used as the input to the local feature encoder ConvNeXt-S. ConvNeXt-S has a total of four layers, each consisting of a convolutional function, which also outputs feature maps at multiple scales. The module names and output sizes are l1: 88x176, l2: 44x88, l3: 22x44, and l4: 11x22, respectively. It runs in parallel with the global feature encoder branch, and the two do not interfere with each other during the encoding stage.

[0131] 3) Input the global features G1, G2, G3, G4 and local features L1, L2, L3, L4 obtained in 2) into the fusion module to obtain the fused feature vectors M1, M2, M3, M4.

[0132] In the formula: concat: concatenation; dim: concatenation dimension; Maxpool: max pooling, size 3, stride 1, padding 1; Avgpool: mean pooling, size 3, stride 1, padding 1; Conv: convolution, size 3, stride 1, padding 1; i: the i-th feature vector.

[0133] The dimensions of features M1, M2, M3, and M4 are 88x176, 44x88, 22x44, and 11x22, respectively.

[0134] 4) Then, the fused feature vectors M1, M2, M3, and M4 obtained in 3) are combined using convolution and bilinear interpolation to finally output a depth map of size 352x704. The specific steps are as follows:

[0135] 4.1) First, merge the fusion features to obtain Y1, Y2 and Y3, with sizes of 22x44, 44x88 and 88x176 respectively.

[0136] 4.2) Upsample Y3 to half the resolution of IuputRGB, 176x352, using the Bilinear scaling function. Then, a convolutional network is used to enhance the network's fitting ability, and finally, Sigmoid normalization is applied.

[0137] 4.3) To obtain the absolute depth map Depth_pre, we need to multiply the normalized Depth_pre_norm obtained in step 4.2) with max_depth, according to the maximum depth value to be predicted that we have set in advance. In order not to increase the computational power, we use bilinear interpolation to restore the absolute depth map with a half-ratio of 176x352 to the size of the original input IuputRGB.

[0138] 4.4) We use mirror prediction for Depth_pre to obtain the final Depth_final.

[0139] Application Example 1:

[0140] This application example uses the image depth estimation method based on global and local features from Example 1. Its effectiveness is verified using the commonly used outdoor dataset KITTI and indoor dataset NYU_v2 for monocular depth estimation tasks. Furthermore, the training model for NYU_v2 is tested on SUNRGBD to verify its generalization performance. Experimental results are shown in Tables 1, 2, and 3. Visualization results on KITTI are shown below. Figure 5 As shown.

[0141] As shown in Table 1 below, the comprehensive evaluation index δ i As can be seen from the meaning of Abs_rel, this method shows significant performance on KITTI, especially in terms of threshold indices δ1, δ2, and δ3, which are improved from 0.702, 0.898, and 0.967 to 0.968, 0.996, and 0.999 respectively compared to existing techniques. The relative absolute error Abs_rel is reduced from 0.203 to 0.055. Compared to the Adabins method with equivalent computational cost, this invention improves performance on Dh_rel by 11%, fully verifying that the method can extract richer global and local information from color images, contributing to the generation of more accurate depth maps.

[0142] Table 1. Results of this invention on the KITTI dataset.

[0143]

[0144] As shown in Table 2 below, combined with the evaluation index δ iAs can be seen from the meaning of Abs_rel, this method not only shows significant performance on the bright KITTI dataset but also provides substantial performance improvements on the relatively dim NYU_v2 dataset. Compared to recent methods, δ1, δ2, and δ3 have improved from 0.769, 0.950, and 0.988 to 0.912, 0.987, and 0.997 respectively, while Abs_rel has decreased from 0.158 to 0.098. Compared to the Adamins method, this invention achieves a 14% performance improvement on Dh_rel, fully validating the effectiveness of the method.

[0145] Table 2 Results of this invention on the NYU_v2 dataset

[0146]

[0147] As shown in Table 3 below, combined with the evaluation index δ i As the meaning of Abs_rel is known, to verify the effectiveness of this method and demonstrate its generalization performance, we tested the model trained on NYU_v2 on SUNRGBD with a test set of 5050 images. Compared with recent methods, δ1, δ2, and δ3 improved from 0.757, 0.943, and 0.984 to 0.808, 0.963, and 0.990 respectively, while Abs_rel decreased from 0.166 to 0.146. Compared with the Adabins method, this invention improves performance on Dh_rel by 4%, fully verifying the effectiveness of the proposed method.

[0148]

[0149] Example 2

[0150] Secondly, this embodiment provides an image depth estimation device based on global and local features, including a processor and a storage medium;

[0151] The storage medium is used to store instructions;

[0152] The processor is configured to operate according to the instructions to perform the steps of the method according to Embodiment 1.

[0153] Example 3

[0154] Thirdly, this embodiment provides a storage medium on which a computer program is stored, which, when executed by a processor, implements the steps of the method described in Embodiment 1.

[0155] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0156] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0157] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0158] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0159] As is known from common technical knowledge, this invention can be implemented through other embodiments that do not depart from its spirit or essential characteristics. Therefore, the disclosed embodiments described above are merely illustrative in all respects and are not the only ones. All modifications within the scope of this invention or its equivalents are included in this invention.

Claims

1. A method for image depth estimation based on global and local features, the method comprising: The method includes: Obtain the color image to be estimated; The color image is preprocessed to obtain the input image InputRGB, including: data preprocessing using the CutDepth-Mask data augmentation method; in order to eliminate missing values, a mask is added to the original CutDepth method, that is, pixels with missing depth values ​​in the original region are retained. Extracting global and local features from the input image InputRGB: The SwinTransformer-T module is used to extract the global features of the input image InputRGB, and the ConvNeXt-S module is used to extract the local features of the input image InputRGB. The extracted global features and local features are fused to obtain a fused feature vector; The fused feature vector is decoded and restored to the size of the input image (InputRGB) to obtain the absolute depth map (Depth_pre). The absolute depth map (Depth_pre) is then mirrored to obtain the predicted depth map (Depth_final), which is used as the image depth estimation result. The process involves using the SwingTransformer-T module to extract global features from the input image (InputRGB). This includes: using the input image (InputRGB) as input to the SwingTransformer-T module; flattening the four-dimensional input variable B x 3 x H x W into a three-dimensional variable B x P x C using a convolutional layer; feeding the three-dimensional variable into a global feature extraction module consisting of four self-attention layers and three downsampling layers for encoding. The self-attention layers consist of multiple linear layers and different activation functions, while the downsampling layers consist of a convolutional layer with a size of 3 and a stride of 2; and the SwingTransformer-T module outputs four global feature vectors G1, G2, G3, and G4 at different scales, representing the dimensions of the input image (InputRGB). The ConvNeXt-S module is used to extract local features from the input image InputRGB. This includes: using the input image InputRGB as input to the ConvNeXt-S module, encoding the InputRGB using a module consisting of four convolutional blocks and four downsampling layers. Each convolutional block consists of one 7x7 convolutional kernel and two 1x1 convolutional kernels cascaded together, outputting four local feature vectors L1, L2, L3, and L4 at different scales, representing the size of the input image InputRGB.

2. The method of claim 1, wherein, The data preprocessing also includes: random horizontal flipping, random cropping, random brightness contrast, random gama regularization, and random saturation. 3.The global and local feature based image depth estimation method of claim 1, wherein, The global features of the input image InputRGB are extracted using the SwingTransformer-T module, including: The input image InputRGB is used as input to the SwinTransformer-T module. A convolutional layer flattens the four-dimensional input variable B x 3 x H x W into a three-dimensional variable B x P x C. The number of convolutional kernels is C x 3, the kernel size is 7, the stride is 4, and the padding is 2. The process is as follows: In the formula, The feature vector generated by the i-th convolutional layer after input image I; The weight parameters of the i-th convolutional kernel; The bias parameter of the i-th convolution kernel; *: convolution operation; the maximum value of i is C; The three-dimensional variable I emb is fed into the global feature extraction module for encoding, and the process of the self-attention layer is shown in the following formula: Q = (I emb • W q + b q ) K = (I emb • W k + b k ) V = (I emb • W v + b v ) In the formula, Q, K, and V are three linear functions; W q W k W v These represent the weight parameters of the three linear layers q, k, and v, respectively; b q b k b v q, k, and v represent the bias parameters of the three linear layers, respectively; softmax represents the logistic regression function, as shown in the formula below: d: Dimension of feature vector K; G i : The global feature vector output by the i-th self-attention layer; The weight parameters of the first linear layer of the i-th self-attention layer; The bias parameters of the first linear layer of the i-th self-attention layer; The weight parameters of the second linear layer in the i-th self-attention layer; The bias parameters of the second linear layer in the i-th self-attention layer; · represents matrix multiplication, and T represents the transpose matrix; The SwinTransformer-T module outputs four different full-scale feature vectors G1, G2, G3, G4 of different scales, which are respectively 4.The global and local feature based image depth estimation method of claim 1, wherein, The extracted global features and local features are fused, including: The local features L1, L2, L3, L4 and the global features G1, G2, G3, G4 are input into the fusion module and fused to obtain the fused feature vectors M1, M2, M3, M4: In the formula: concat: concatenation; dim: concatenation dimension; Maxpool: max pooling, size 3, stride 1, padding 1; Avgpool: mean pooling, size 3, stride 1, padding 1; Conv: convolution, size 3, stride 1, padding 1; i: the i-th feature vector. These are intermediate parameters.

5. The image depth estimation method based on global and local features according to claim 1, characterized in that, The fused feature vectors are decoded and restored to the InputRGB size to obtain the absolute depth map Depth_pre, which includes: The fused feature vectors M1, M2, M3, and M4 are merged to obtain the merged fused feature vectors Y1, Y2, and Y3. Two consecutive convolutions are used to connect adjacent fused feature vectors M1, M2, and M3. i M i+1 : Y1=Conv(Conv(concat(bilinear(M1),M2,dim=1))) Y2=Conv(Conv(concat(bilinear(Y1),M3,dim=1))) Y3=Conv(Conv(concat(bilinear(Y2),M4,dim=1))) In the formula, bilinear: bilinear interpolation, scaling ratio 2; dim: dimension of splicing; concat: splicing; Conv: convolution, Y i : feature map after each scaling and merging Y3 is scaled to half the resolution of the input image (InputRGB) using the Bilinear scaling function. A convolutional network is used to enhance the network's fitting ability, and finally, Sigmoid normalization is applied to obtain the normalized Depth_pre_norm. The process is shown in the following formula: Depth_pre_norm=Sigmoid(Conv(bilinear(Y3))) The normalized Depth_pre_norm is multiplied by the preset maximum depth value max_depth, and then bilinear interpolation is used to restore it to the size of the input image InputRGB, thus obtaining the absolute depth map Depth_pre.

6. The image depth estimation method based on global and local features according to claim 1, characterized in that, The predicted depth map Depth_final is obtained by mirror prediction of the absolute depth map Depth_pre, including: the absolute depth map Depth_pre includes the left image Depth_pre_L and the right image Depth_pre_R of the absolute depth map; In the formula, "ship" means horizontal flipping.

7. An image depth estimation device based on global and local features, characterized in that, Including processor and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1 to 6.

8. A storage medium having stored thereon a computer program, characterized in that When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images