A monocular image depth estimation method based on hybrid neural network model

Abstract

The present invention belongs to the field of computer vision and discloses a monocular image depth estimation method based on a hybrid neural network model, comprising: given an RGB image and a true depth value, performing a normalization operation, passing the RGB image through a convolution module to obtain shallow image features, then dividing the features into blocks, performing feature enhancement of different regions, obtaining features with importance discrimination, and passing the features through a Transformer module in sequence to obtain deep features, combining the shallow image features with the deep features to form an image feature sequence, performing feature fusion, and obtaining a complete feature representation, then outputting a block using depth information to obtain a depth map, performing error analysis between the depth map and the true depth value, obtaining a final network model, and completing depth estimation. The present invention establishes local information and long-range dependency relationships, enhances the capture of spatial features and the expression of local structures, and achieves more accurate depth estimation results.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and specifically relates to a monocular image depth estimation method based on a hybrid neural network model. Background Art

[0002] Monocular depth estimation plays a vital role in numerous image or video based applications, including depth-aware view synthesis, augmented reality, and special effects. Most existing methods use variants of convolutional neural networks to achieve a balance between accuracy and computation in MDE. Convolutional neural networks perform well in depth estimation tasks, with a strong ability to capture local and global spatial information, effectively extracting depth-related features from images. However, it is sensitive to the sampling interval when processing images. In the case of irregular sampling intervals between different pixels, especially in edges and texture-rich areas, the estimated depth map often becomes inaccurate. The introduction of visual Transformer provides a self-attention mechanism that enables it to model long-range dependencies. This mechanism allows handling irregular sampling and improves.

[0003] However, existing monocular depth estimation methods have difficulty retaining detailed information when processing areas rich in edges and textures. This is mainly reflected in their high sensitivity to sampling intervals. When there are irregular sampling intervals between different pixels, the feature information captured by areas rich in edges and textures is insufficiently represented, resulting in the generated depth being often inaccurate. In addition, existing methods partially rely on the advantages of a single deep learning framework, ignoring the joint advantages of hybrid frameworks for monocular depth estimation methods, and thus fail to fully establish feature relationships between local and global information, and lack differentiated processing of different areas of the image. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention provides a monocular image depth estimation method based on a hybrid neural network model. This method combines the advantages of hybrid neural networks and uses shallow features and deep features to jointly establish local information and long-range dependencies, thereby enhancing the capture of spatial features and the expression of local structures, and achieving more accurate depth estimation results.

[0005] In order to achieve the above object, the present invention is achieved through the following technical solutions:

[0006] The present invention is a monocular image depth estimation method based on a hybrid neural network model, which comprises the following steps:

[0007] Step 1: Given an RGB image and its corresponding true depth value;

[0008] Step 2: Normalize the given RGB image to make it meet the input requirements of the convolution module in step 3;

[0009] Step 3: The standardized RGB image passes through two convolution modules in sequence to obtain shallow image features;

[0010] Step 4: Divide the shallow image features obtained in step 3 into blocks, and use the block attention module to enhance the features of different regions to obtain features with importance discrimination;

[0011] Step 5: Pass the features with importance discrimination obtained in step 4 through the Transformer module in sequence to obtain deep features;

[0012] Step 6: The shallow image features obtained in step 3 and the deep features obtained in step 5 are combined into image feature sequences of different scales. The features are fused sequentially from deep to shallow in the feature fusion module to obtain a complete feature representation.

[0013] Step 7: The complete feature representation obtained in step 6 is mapped to the corresponding depth representation using the depth information output block to obtain a depth map;

[0014] Step 8: Use translation invariance loss to perform error analysis on the depth map obtained in step 7 and the true depth value of the RGB image in step 1. Under the condition of minimizing the target loss, optimize the model to obtain the final network model and complete the depth estimation.

[0015] A further improvement of the present invention is that: in step 3, the convolution module is responsible for downsampling and modeling local information, and the convolution modules are taken from the first and second layers of the standard ResNet respectively. For a given input image I∈R H ×W×3 , where (H, W) represents the resolution of the original image. Convolutional modules 1 and 2 are used to gradually downsample the image by 4 and 2 times respectively. The size of the output feature map of convolutional module 1 is [H / 4, W / 4, C], and the size of the output feature map of convolutional module 2 is [H / 8, W / 8, C], where C represents the channel dimension.

[0016] A further improvement of the present invention is that: in step 4, the shallow image features captured by convolution are converted into blocks of different importance through the block attention module, and then sent to the Transformer module to generate more representative labels. The block attention module aggregates shallow image features in the vertical and horizontal directions, captures long-range dependencies, and retains accurate location information. Specifically, the following steps are included:

[0017] Step 4-1: Generate a shallow image feature map by the convolution module and divide it into blocks Two average poolings (ph, 1) or (1, pw) are used to encode each channel along the horizontal and vertical coordinates, respectively, to obtain:

[0018]

[0019]

[0020] in, and They represent the output of the c-th channel at height ph and width pw, respectively, i is the i-th feature block in the vertical direction, j is the j-th feature block in the horizontal direction, and p is the number of feature blocks;

[0021] Step 4-2, and The output of is connected and then transformed by convolution:

[0022] f=δ(F1([z ph , z pw ])

[0023] Among them, [z ph , z pw ] represents the connection operation along the spatial dimension, δ is the nonlinear activation function, and f is the intermediate feature map;

[0024] Step 4-3: Split f into two separate sequences f along the spatial dimension ph and f pw At the same time, the other two 1×1 convolution transformations F ph and F Pw is used to separate f ph and f pw Convert to a sequence with the same number of channels as the input block.

[0025] g ph =σ(F ph (f ph ))

[0026] g pw =σ(F pw (f pw ))

[0027] Among them, σ represents the sigmoid function, f ph and f pw are two separate sequences, F ph and F Pw are two 1×1 convolution transformations, g ph and g pw There are two weights.

[0028]

[0029] Where x′ c (i, j) is a weighted block, which is then embedded into the token for further processing, x c (i, j) represents the input block, and are two spatial weights.

[0030] A further improvement of the present invention is that: Step 5 is specifically: following the Transformer module, using CNN Block1 to split it into non-overlapping blocks x∈R H / 4p×W / 4p×C , where (p, p) represents the resolution of each feature block. Non-overlapping blocks are first strengthened by the block attention module, then flattened into a sequence, and embedded into a space of dimension D through linear projection. Then, position embedding is added as the input of the Transformer module, denoted as t 0 , each layer consists of a multi-head self-attention (MSA) mechanism module and a multi-layer perceptron (MLP) module. Layer normalization (LN) is applied before the multi-head self-attention (MSA) mechanism module and the multi-layer perceptron (MLP) module, and a residual connection is used for each module. The specific formula is as follows:

[0031]

[0032]

[0033]

[0034] in, t0、E pos denote the tag sequence and position embedding respectively, D is the feature dimension of each sequence, and t l They represent the deep features output by the multi-head self-attention (MSA) mechanism module and the deep features output by the multi-layer perceptron (MLP) module in the lth layer respectively.

[0035] A further improvement of the present invention is that: Step 6 is specifically as follows: the deep features obtained in step 5 need to be first passed through the recombination module and then fused with the shallow image features obtained in step 3 to obtain a complete feature representation, wherein the feature representation output by the Transformer module is unified by the recombination module, specifically:

[0036]

[0037]

[0038] Among them, R is the reorganization operation, pro(t) is the projection operation, l is the lth Transformer layer, The label information is passed to each sequence through a linear layer and a GELU nonlinear function to restore the original dimension D, Unflatten(t)∈R h / p×w / p×C Each sequence is placed in the image according to the position information of the original block to obtain an image-like representation, and then Resample(t)∈R H / s×W / s×C Resampling is performed, and s is the spatial scale of the current layer.

[0039] A further improvement of the present invention is that: step 7 is specifically: using a 3×3 convolution module to perform convolution operations on different channels of the depth feature, using an upsampling module to increase the resolution of the feature map so that the size of the feature map is consistent with the original input image, using a ReLu activation function to achieve nonlinear transformation, and using a 1×1 convolution to reduce the dimension of the feature map or adjust the number of channels.

[0040] The beneficial effects of the present invention are:

[0041] Based on the hybrid neural network, the present invention further optimizes the feature conversion strategy between different modules, adds block attention between shallow features and deep features to optimize the capture of deep features, enhances the capture of spatial features and the expression of local structures, establishes long-distance dependencies between pixels, and thus achieves more accurate depth estimation results.

[0042] The present invention uses the block attention mechanism to prioritize the effective representation of features in different regions and then pass them into the transformer module, thereby avoiding the transformer's characteristic of treating different regions equally, strengthening the feature representation of the image, and optimizing the feature interaction of the hybrid structure.

[0043] The present invention implements a block attention mechanism between features across different neural network modules, prioritizes differentiated processing of different regions of shallow features, improves the effective representation of local features, and optimizes the capture of deep features by subsequent Transformer blocks. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is the overall architecture diagram of the monocular image depth estimation method of the present invention.

[0045] Figure 2 Schematic diagram of the block attention module of the present invention.

[0046] Figure 3 is a schematic diagram of the recombination module of the present invention.

[0047] Figure 4 Schematic diagram of the feature fusion module of the present invention. DETAILED DESCRIPTION

[0048] The following diagrams illustrate embodiments of the present invention. For clarity, many practical details are included in the following description. However, it should be understood that these practical details are not intended to limit the present invention. In other words, in some embodiments of the present invention, these practical details are not essential.

[0049] like Figure 1 As shown, the present invention is a monocular image depth estimation method based on a hybrid neural network model, and the monocular image depth estimation method includes the following steps:

[0050] Step 1: Given an RGB image and its corresponding true depth value.

[0051] Step 2: Normalize the given RGB image to make it meet the input requirements of the convolution module in step 3.

[0052] Step 3: The standardized RGB image passes through two convolution modules in sequence to obtain shallow image features.

[0053] In this step, the convolution modules are responsible for downsampling and modeling local information. The convolution modules are taken from the first and second layers of the standard ResNet, respectively. For a given input image I∈R H×W×3 , where (H, W) represents the resolution of the original image. Convolutional modules 1 and 2 are used to gradually downsample the image by 4 and 2 times respectively. The size of the output feature map of convolutional module 1 is [H / 4, W / 4, C], and the size of the output feature map of convolutional module 2 is [H / 8, W / 8, C], where C represents the channel dimension.

[0054] Step 4: Divide the shallow image features obtained in step 3 into blocks, and use the block attention module to enhance the features of different areas to obtain features with importance discrimination.

[0055] Step 4 converts the shallow image features captured by convolution into blocks of different importance through the block attention module, and then feeds them into the Transformer module to generate more representative tags. The block attention module aggregates shallow image features in the vertical and horizontal directions, captures long-range dependencies, and retains accurate location information. Specifically, it includes the following steps:

[0056] Step 4-1: Generate a shallow image feature map by the convolution module and divide it into blocks Two average poolings (ph, 1) or (1, pw) are used to encode each channel along the horizontal and vertical coordinates, respectively, to obtain:

[0057]

[0058]

[0059] in, and They represent the output of the c-th channel at height ph and width pw, respectively, i is the i-th feature block in the vertical direction, j is the j-th feature block in the horizontal direction, and p is the number of feature blocks;

[0060] Step 4-2, and The output of is connected and then transformed by convolution:

[0061] f=δ(F1([z ph , z pw ])

[0062] Among them, [z ph , z pw ] represents the connection operation along the spatial dimension, δ is the nonlinear activation function, and f is the intermediate feature map;

[0063] Step 4-3: Split f into two separate sequences f along the spatial dimension ph and f pw At the same time, the other two 1×1 convolution transformations F ph and F Pw is used to separate f ph and f pw Convert to a sequence with the same number of channels as the input block.

[0064] g ph =σ(F ph (f ph ))

[0065] g pw =σ(F pw (f pw ))

[0066] Among them, σ represents the sigmoid function, f ph and f pw are two separate sequences, F ph and F Pw are two 1×1 convolution transformations, g ph and g pw There are two weights.

[0067]

[0068] Where x′ c (i, j) is a weighted block, which is then embedded into the token for further processing, x c(i, j) represents the input block, and are two spatial weights.

[0069] like Figure 2 As shown, the block attention module of the present invention first encodes each channel along the horizontal coordinate and the vertical coordinate through two average pooling modules, and then transforms the output sequence of the average pooling module through a connected convolution module. The sequence is then normalized and nonlinearly processed to segment the sequence, and the number of channels of the sequence is transformed by convolution to make it consistent with the channels of the input sequence. The activation function is then used to multiply and further multiply the original input sequence to obtain the weighted feature result.

[0070] Step 5: Pass the important features obtained in step 4 through the Transformer module in sequence to obtain deep features. Specifically, follow the Transformer module design and use CNN Block 1 to split it into non-overlapping blocks x∈R H / 4p×W / 4p×C , where (p, p) represents the resolution of each feature block. Non-overlapping blocks are first strengthened by the block attention module, then flattened into a sequence, and embedded into a space of dimension D through linear projection. Then, position embedding is added as the input of the Transformer module, denoted as t 0 , each layer consists of a multi-head self-attention (MSA) mechanism module and a multi-layer perceptron (MLP) module. Layer normalization (LN) is applied before the multi-head self-attention (MSA) mechanism module and the multi-layer perceptron (MLP) module, and a residual connection is used for each module. The specific formula is as follows:

[0071]

[0072]

[0073]

[0074] in, t0、E pos denote the tag sequence and position embedding respectively, D is the feature dimension of each sequence, and t l They represent the deep features output by the multi-head self-attention (MSA) mechanism module and the deep features output by the multi-layer perceptron (MLP) module in the lth layer respectively.

[0075] Step 6: Combine the shallow image features obtained in step 3 and the deep features obtained in step 5 to form image feature sequences of different scales. In the feature fusion module, feature fusion is performed from deep to shallow to obtain a complete feature representation.

[0076] Step 6 is as follows: The deep features obtained in step 5 need to be first passed through the reorganization module and then fused with the shallow image features obtained in step 3 to obtain a complete feature representation. The feature representation output by the Transformer module is unified by the reorganization module, specifically:

[0077]

[0078]

[0079] Among them, R is the reorganization operation, pro(t) is the projection operation, l is the lth Transformer layer, The label information is passed to each sequence through a linear layer and a GELU nonlinear function to restore the original dimension D, Unflatten(t)∈R h / p×w / p×C Each sequence is placed in the image according to the position information of the original block to obtain an image-like representation, and then Resample(t)∈R H / s×W / s×C Resampling is performed, and s is the spatial scale of the current layer.

[0080] like Figure 3 The reorganization module shown maps the label information sequence to each sequence, restores the original dimension, then returns the sequence to the original image position through the Unflatten operation, and then resamples to restore the original scale.

[0081] like Figure 4 The feature fusion module shown here consists of two residual convolutional layers, an upsampling module, and a projection module. One residual convolutional layer processes shallow features, adds them to deep features, and feeds them into another residual convolutional layer, followed by upsampling and projection. The residual convolutional layer consists of a ReLu activation function layer, a convolution module, a batch normalization module, a ReLu activation function layer, a convolution module, and a batch normalization module. The convolution kernels of both convolutional modules are 3×3.

[0082] Step 7: The complete feature representation obtained in step 6 is mapped to the corresponding depth representation using the depth information output block to obtain the depth map. Specifically, a 3×3 convolution module is used to perform convolution operations on different channels of the depth feature, and an upsampling module is used to increase the resolution of the feature map so that the size of the feature map is consistent with the original input image. The ReLu activation function is used to implement nonlinear transformation, and a 1×1 convolution is used to reduce the dimension of the feature map or adjust the number of channels.

[0083] Step 8: Use translation invariance loss to perform error analysis on the depth map obtained in step 7 and the true depth value of the RGB image in step 1. Under the condition of minimizing the target loss, optimize the model to obtain the final network model and complete the depth estimation.

[0084] This method combines the advantages of hybrid neural networks, using shallow features and deep features to jointly establish local information and long-distance dependencies, enhancing the capture of spatial features and the expression of local structures, and achieving more accurate depth estimation results.

[0085] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A method for monocular image depth estimation based on a hybrid neural network model, characterized by: The monocular image depth estimation method comprises the following steps: Step 1: Given an RGB image and its corresponding true depth value; Step 2: Normalize the given RGB image to make it meet the input requirements of the convolution module in step 3; Step 3: The standardized RGB image passes through two convolution modules in sequence to obtain shallow image features; Step 4: Divide the shallow image features obtained in step 3 into blocks, and use the block attention module to enhance the features of different regions to obtain features with importance discrimination; Step 5: Pass the features with importance discrimination obtained in step 4 through the Transformer module in sequence to obtain deep features; Step 6: The shallow image features obtained in step 3 and the deep features obtained in step 5 are combined into image feature sequences of different scales. The features are fused sequentially from deep to shallow in the feature fusion module to obtain a complete feature representation. Step 7: The complete feature representation obtained in step 6 is mapped to the corresponding depth representation using the depth information output block to obtain a depth map; Step 8: Perform error analysis on the depth map obtained in step 7 and the true depth value of the RGB image in step 1 using translation invariance loss. Under the condition of minimizing the target loss, optimize the model to obtain the final network model and complete the depth estimation, where: Step 4 converts the shallow image features captured by convolution into blocks of different importance through the block attention module, and then feeds them into the Transformer module to generate more representative tags. The block attention module aggregates shallow image features in the vertical and horizontal directions, captures long-range dependencies, and retains accurate location information. Specifically, it includes the following steps: Step 4-1: Generate a shallow image feature map by the convolution module and divide it into blocks Use two average poolings (ph,1) or (1,pw) to encode each channel along the horizontal and vertical coordinates, respectively, to obtain: in, and They represent the output of the c-th channel at height ph and width pw, respectively, i is the i-th feature block in the vertical direction, j is the j-th feature block in the horizontal direction, and p is the number of feature blocks; Step 4-2, and The output of is connected and then transformed by convolution: f=δ(F1([z ph ,z pw ]) Among them, [z ph ,z pw ] represents the connection operation along the spatial dimension, δ is the nonlinear activation function, and f is the intermediate feature map; Step 4-3: Split f into two separate sequences f along the spatial dimension ph and f pw At the same time, the other two 1×1 convolution transformations F ph and F Pw is used to separate f ph and f pw Converted to a sequence with the same number of channels as the input block, g ph =σ(F ph (f ph )) g pw =σ(F pw (f pw )) Among them, σ represents the sigmoid function, f ph and f pw are two separate sequences, F ph and F Pw are two 1×1 convolution transformations, g ph and g pw are two weights, Where x′ c (i,j) is a weighted block, which is then embedded into the token for further processing, x c (i,j) represents the input block, and are two spatial weights.

2. The method for monocular image depth estimation based on a hybrid neural network model according to claim 1, wherein: In step 3, the convolution module is responsible for downsampling and modeling local information. The convolution modules are taken from the first and second layers of the standard ResNet, respectively. For a given input image I∈R H×W×3 , where (H, W) represents the resolution of the original image. Convolutional modules 1 and 2 are used to gradually downsample the image by 4 and 2 times respectively. The size of the output feature map of convolutional module 1 is [H / 4, W / 4, C], and the size of the output feature map of convolutional module 2 is [H / 8, W / 8, C], where C represents the channel dimension.

3. The method for monocular image depth estimation based on a hybrid neural network model according to claim 1, wherein: The step 5 is specifically as follows: follow the Transformer module and use CNN Block 1 to split it into non-overlapping blocks x∈R H / 4p×W / 4p×C , where (p, p) represents the resolution of each feature block. Non-overlapping blocks are first strengthened by the block attention module, then flattened into a sequence, and embedded into a space of dimension D through linear projection. Then, position embedding is added as the input of the Transformer module, denoted as t 0 , each layer consists of a multi-head self-attention (MSA) mechanism module and a multi-layer perceptron (MLP) module. Layer normalization (LN) is applied before the multi-head self-attention (MSA) mechanism module and the multi-layer perceptron (MLP) module, and a residual connection is used for each module. The specific formula is as follows: in, t0、E pos denote the tag sequence and position embedding respectively, D is the feature dimension of each sequence, and t l They represent the deep features output by the multi-head self-attention (MSA) mechanism module and the deep features output by the multi-layer perceptron (MLP) module in the lth layer respectively.

4. The method for monocular image depth estimation based on a hybrid neural network model according to claim 1, wherein: The step 6 is specifically as follows: the deep features obtained in step 5 need to be first passed through the recombination module and then fused with the shallow image features obtained in step 3 to obtain a complete feature representation, wherein the feature representation output by the Transformer module is unified by the recombination module, specifically: R=Resample о Unflatten о Pro(t) Among them, R is the reorganization operation, pro(t) is the projection operation, l is the lth Transformer layer, The label information is passed into each sequence through a linear layer and a GELU nonlinear function to restore the original dimension D. Unflatten(t)∈R h / p×w / p×C Each sequence is placed in the image according to the position information of the original block to obtain an image-like representation, and then Resample(t)∈R H / s×W / s×C Resampling is performed, and s is the spatial scale of the current layer.

5. The method for monocular image depth estimation based on a hybrid neural network model according to claim 1, wherein: The step 7 is specifically as follows: using a 3×3 convolution module to perform convolution operations on different channels of the depth feature, using an upsampling module to increase the resolution of the feature map so that the size of the feature map is consistent with the original input image, using a ReLu activation function to achieve nonlinear transformation, and using a 1×1 convolution to reduce the dimension of the feature map or adjust the number of channels.

Images