A monocular image depth estimation method and storage medium
By using the triple cross attention TCA module and iterative optimization network in monocular image depth estimation, the information loss problem caused by the connection between mobile window capture windows is solved, and a more accurate and robust depth estimation effect is achieved.
Patent Information
- Application Number
- CN202411210367.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-30
AI Technical Summary
Existing monocular image depth estimation technology leads to loss of image information when using mobile windows to capture the connection between windows, making it difficult to take into account local, global and correlation information, resulting in poor depth estimation effect.
The triple cross attention TCA module is adopted, including window attention WCA, cross-dimensional attention CDA and correlation attention REA module, combined with feature block selection and fusion FBSF module, through window attention, cross-dimensional attention and correlation attention Forces to fusion of information, generate more robust feature representations, and optimize preliminary depth estimation results through iterative optimization of networks.
It realizes a more comprehensive extraction of image information, solves the problem of information loss, improves the accuracy and robustness of depth estimation, and improves the prediction accuracy especially when the colors are close but the depth is different.
Smart Images

Figure CN118735978B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing, and in particular to a monocular image depth estimation method and a storage medium. Background Art
[0002] Depth estimation of monocular images is a fundamental research problem in the field of computer vision. The task aims to predict and restore pixel-level depth information of a scene based on a single image. Monocular depth estimation technology is a key supporting technology for high-level visual tasks such as 3D reconstruction, environmental perception and understanding, and has a wide range of application needs. In recent years, with the rapid development of deep learning technology, monocular depth estimation has also made great progress. Early deep learning-based monocular depth estimation models were usually based on convolutional neural networks (CNNs). Given the powerful feature extraction and representation capabilities of deep neural networks, CNN-based methods have achieved better depth estimation results than traditional methods. Since then, most of the work has been devoted to designing increasingly complex and powerful networks to improve model performance. However, without additional information guidance, the depth estimation task has become difficult to fit.
[0003] Recently, Transformer has become popular in the field of natural language processing. Researchers have introduced this attention-based network structure into the field of computer vision and achieved good results. In the field of monocular depth estimation, the Transformer architecture has also shown excellent results in recent years. It captures long-distance dependencies of features through self-attention. While achieving superior performance, it is often accompanied by high computational complexity. The strategy of combining window self-attention and moving windows effectively reduces the complexity of the algorithm, but it is difficult to perceive global information. Summary of the invention
[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a monocular image depth estimation method and storage medium to solve the problem of image information loss caused by the connection between windows captured by a moving window in the prior art, take into account local, global and correlation information, realize information exchange between windows and between windows and the entire image, extract image information more comprehensively and achieve more accurate depth estimation effect.
[0005] In order to solve the above technical problems, the present invention is implemented by adopting the following technical solutions:
[0006] In a first aspect, the present invention provides a monocular image depth estimation method, which is characterized by comprising:
[0007] Obtain the monocular image to be estimated and preprocess it;
[0008] Input the preprocessed image into a pre-trained preliminary depth estimation network to obtain a preliminary depth estimation result of the image;
[0009] Inputting the preliminary depth estimation result into a pre-trained iterative optimization network to obtain a final monocular image depth estimation result;
[0010] The preliminary depth estimation network includes an encoder, a decoder and a regression head;
[0011] The encoder includes a plurality of sequentially connected encoding layers, and the decoder includes a plurality of sequentially connected decoding layers, wherein each encoding layer corresponds one-to-one with each decoding layer in the reverse direction of the decoding order along the encoding order; the decoding layer adopts a triplet cross attention TCA module,
[0012] The triplet cross attention TCA module includes a window attention WCA module, a cross-dimensional attention CDA module, a correlation attention REA module and a feature block selection fusion FBSF module;
[0013] The window attention WCA module is used to calculate the attention of the input encoding features and decoding features, and output the window attention features;
[0014] The cross-dimensional attention CDA module is used to calculate the attention of the input decoding features and the window attention features, and output the cross-dimensional attention features;
[0015] The correlation attention REA module is used to extract the window correlation feature of the window attention feature and calculate the attention, and output the correlation attention feature;
[0016] The feature block selection fusion FBSF module fuses the window attention feature, the cross-dimensional attention feature and the correlation attention feature to generate the output feature of the decoding layer;
[0017] The output of the last decoding layer is the final decoding feature; the encoding feature is the output feature of the encoding layer corresponding to the current decoding layer; if the current decoding layer is the first decoding layer, the decoding feature input to the current decoding layer is the output feature after the output features of each encoding layer are fused; if the current decoding layer is not the first decoding layer, the decoding feature input to the current decoding layer is the output feature of the previous decoding layer;
[0018] The regression head is used to generate the preliminary depth estimation result according to the final decoded feature.
[0019] In the above-mentioned image depth estimation process, the window attention WCA, cross-dimensional attention CDA and correlation attention REA modules are designed to characterize the feature relationship between local, global and local features, and the feature block selection fusion FBSF module is designed to effectively fuse the attention under three different perspectives, remove highly similar feature representations, and take into account the correlation of local, global information and local information to obtain a more robust feature representation. The preliminary depth estimation result generated by the iterative optimization module is optimized to obtain the final monocular image depth estimation result.
[0020] Optionally, the window attention WCA module calculates attention on the input encoding features and decoding features including:
[0021] S11: splitting the preprocessed image into windows;
[0022] S12: In each window, calculate the Q vector for the encoding feature and K vector , calculate the V vector for the decoded features , where Q vector, K vector and V vector are respectively the Quary matrix, Key matrix and Value matrix obtained by linearly changing the corresponding features;
[0023] S13: Q vector calculated based on each window , K vector and V vector , calculate the window attention output as the window attention feature, the formula is:
[0024] ,
[0025] in, represents the window attention feature, represents the normalization function, b is the relative position deviation matrix, and d represents the Q vector and K vector The ratio of the vector dimensions of b, each element in b represents the relative position embedding between position m and position n, where m represents the Q vector The element position in, n represents the K vector The position of the element in .
[0026] The window attention WCA module is used to focus on the detailed features within the local window of the image. The relative position encoding uses a learnable matrix b to represent the relative position between the query Q and the key K pair. In this way, when calculating the attention score, not only the content similarity is considered, but also the relative position information is added. After calculating the attention within each window, the results are rearranged according to the original position to obtain the final attention output. This method can improve the expressiveness and generalization ability of the model while capturing the position information, and better focus on the detailed features within the local window of the image.
[0027] Optionally, the preliminary depth estimation network further includes a pyramid pooling PPM module, which receives the output features of each of the encoding layers and performs fusion processing to obtain the decoding features input into the first decoding layer.
[0028] Optionally, the cross-dimensional attention CDA module includes a global vector generator GVG, and the global vector generator GVG is used to receive the decoded feature processing to generate a global perception vector, and the process is:
[0029] S211: compressing the size of the dimension of the decoding feature to be the same as the window dimension, wherein the size of the dimension of the decoding feature and the size of the window dimension are respectively the decoding feature resolution and the window resolution;
[0030] S212: copying the compressed decoding feature to restore the dimension of the original decoding feature, and supplementing the edge with a mask;
[0031] S213: Extracting a V vector from the compressed decoding features of each window size of the restored decoding features as a global perception vector.
[0032] When the compressed decoded features are copied and restored to the original decoded feature dimensions, if the edge is not large enough to place the entire compressed decoded feature, a mask is used to complete it. The mask is a binary image with only values 0 and 1, which is used to identify which pixels are valid, that is, the pixels in the original feature, and which pixels are filled or copied, that is, the pixels that need to be masked.
[0033] Optionally, the cross-dimensional attention CDA module is used to calculate the attention of the input decoding features and the window attention features, including:
[0034] S21: The global vector generator GVG generates the global perception vector, the global perception vector is used as the V vector, and the window attention feature is calculated to obtain the Q vector and the K vector;
[0035] S22: Calculate attention on the Q vector, K vector and V vector, and output the cross-dimensional attention feature.
[0036] The global perception vector represents a longer-distance or global long-range dependency. The global perception vector is used as a V vector containing global information, and together with the local feature vector, it participates in the attention calculation between different dimensions within the window to obtain a cross-dimensional attention feature, which can represent the relationship between the entire image and the current window feature, and realize the interaction of information inside and outside the window.
[0037] Optionally, the correlation attention REA module is used to extract window correlation features from the window attention features and calculate attention, including:
[0038] S31: Split the window attention feature into different windows, and each window is linearly transformed to obtain a Q vector , K vector and V vector ;
[0039] S32: Q vector and K vector Take the average value in each window as the mark of the window;
[0040] S33: Extract the Q vectors respectively and K vector The flags of each window correspond to the generation of the intermediate Q vector matrix and the intermediate K vector matrix ;
[0041] S34: The intermediate Q vector matrix and the intermediate K vector matrix Multiply the transposed matrix of to generate an adjacency matrix representing the affinity between windows;
[0042] S35: constructing an index function based on the adjacency matrix to obtain which k windows each window has the highest correlation with, where k is a non-zero natural number less than the number of windows;
[0043] S36: respectively performing the operations on the K vectors and V vector Use the index function to index and use the collection function to collect the corresponding index results to generate a K vector containing window correlation information and V vector containing window correlation information ;
[0044] S37: Q vector , K vector containing window correlation information and V vector containing window correlation information Calculate attention and output relevant attention features.
[0045] The relevance attention REA module uses semantic relevance to calculate the attention between highly correlated windows, realizes information exchange between windows, and thus obtains the global dependency of each window.
[0046] Optionally, the feature block selects a fusion FBSF module to fuse the window attention feature, the cross-dimensional attention feature and the correlation attention feature, including:
[0047] S41: Set the sum of the cross-dimensional attention feature and the correlation attention feature as the first feature G u ;
[0048] S42: The first feature G u The window attention feature is concatenated along the channel dimension to form the second feature B u ;
[0049] S43: The second feature B u Split into feature blocks through slicing operation;
[0050] S44: The segmented feature blocks are selected by interval selection operation, and the maximum value in each selected feature block is extracted to generate the third feature B u ';
[0051] S45: The third feature B u 'Apply convolution operation to generate two channels, apply sigmoid function to each channel to generate a two-channel attention map, and convert the first feature G u The window attention feature is multiplied with the dual-channel attention map to generate the fourth feature G u ' and the fifth characteristic L u ';
[0052] S46: The fourth feature G u ' and the fifth characteristic L u 'Add together to generate the output features of this decoding layer.
[0053] The feature block selection fusion FBSF module effectively fuses the attention from three different perspectives. u It is composed of the sum of cross-dimensional attention and correlation attention results. These features have global long-range dependencies. uIt is a local feature obtained through the window self-attention mechanism, which has strong local correlation. Through the feature block selection operation, the highly similar feature representations are removed, which can reduce the dimension of the feature while retaining the key information, thereby obtaining more representative features. Finally, a dual-channel attention map is generated through the sigmoid operation, in which each local and global feature is multiplied by a channel map to focus on important positions and obtain the final feature representation.
[0054] Optionally, the iterative optimization network includes an iterative optimizer based on a gated recurrent unit GRU, and the iterative optimization network obtains a final monocular image depth estimation result based on a preliminary depth estimation result, including:
[0055] S51: Computing Uncertain Graphs And the depth map of stage t The difference between the two represents the difference map at stage t , the formula is:
[0056] = ,
[0057] Among them, when t=1, is the initial depth map, whose value is the preliminary depth estimation result, and the uncertainty map Generated by the final decoded features through the regression head;
[0058] S52: The depth map of the tth stage , Uncertain Graph , the difference map of stage t , the hidden features of the tth stage and the initial hidden features Input gated recurrent unit GRU generates hidden features of the t+1th stage , the specific calculation is:
[0059] ,
[0060] in, is the output feature of the first encoding layer;
[0061] S53: The hidden features of the t+1th stage The calculation result generated by the input update module △ is superimposed on the depth map of the tth stage , output the depth map of the t+1th stage , the specific calculation is:
[0062] = +△( ),
[0063] Among them, the update module △ is a Conv-ReLU combination operation,
[0064] The iterative process of the iterative optimizer is as follows: set the number of iterations to x, where x is a non-zero natural number, and the initial value of t is 1, and sequentially execute steps S51-S53 in a loop until t is equal to x. is the final monocular image depth estimation result.
[0065] The preliminary depth estimation results generated by the present invention have significantly improved the effect of object edges and contours. However, in some pictures, for those with similar colors but different depths, the prediction of depth changes is wrong. In order to further improve the prediction accuracy, the present invention uses an iterative optimizer based on GRU to iteratively optimize the preliminary depth estimation results.
[0066] Optionally, training the preliminary depth estimation network and the iterative optimization network includes training a loss function, and the adopted loss function includes uncertainty loss and depth map loss;
[0067] Calculating the uncertainty loss includes:
[0068] S61: Use the probability density function of the Laplace distribution to construct the uncertainty truth value of the i-th pixel :
[0069] ,
[0070] in, The image depth truth value of the i-th pixel provided by the dataset, is the image depth prediction value of the i-th pixel, the image depth prediction value is the depth map of each stage of the iterative optimization network, T represents the coefficient of control error tolerance, and exp is an exponential function with e as the base;
[0071] S62: Calculating uncertainty loss :
[0072] ,
[0073] in, is the uncertainty value of the uncertainty map of the i-th pixel, indicating the image uncertainty prediction value of the i-th pixel, It is the sum operation of i absolute values;
[0074] Calculating the depth map loss includes:
[0075] S71: Calculate the depth prediction value of the i-th pixel in the image And the depth truth The difference in logarithmic functions between them is:
[0076] ,
[0077] S72: Calculate the depth map loss SILog as follows:
[0078] ,
[0079] Where y is the total number of pixels, is the variance minimization factor, is the scale constant;
[0080] Overall loss Calculate results for SILog Loss with uncertainty sum:
[0081] .
[0082] In order to mitigate the negative impact of depth noise, an uncertainty correction loss is introduced to reduce the negative impact of high uncertainty areas on uncertainty map generation by correcting the parts with high uncertainty. At the same time, SILog is used to supervise the depth maps generated at each stage of the iterative optimizer. The uncertainty loss is superimposed on the depth map loss to form the overall loss function training model to achieve more accurate depth estimation.
[0083] In a second aspect, the present invention provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, enable the computing device to perform a monocular image depth estimation method.
[0084] Compared with the prior art, the present invention has the following beneficial effects:
[0085] The present invention is suitable for monocular image depth estimation, which solves the problem of image information loss caused by the connection between windows captured by the prior art using a moving window. It takes into account local, global and correlation information, realizes information exchange between windows and between the window and the entire image, and extracts image information more comprehensively to achieve more accurate depth estimation effect.
[0086] Specifically, the window attention WCA module is designed to focus on the detailed features within the local window of the image. The cross-dimensional attention CDA module realizes the interaction of information inside and outside the window by calculating the relationship between the entire image and the current window features. The correlation attention REA module uses semantic correlation to calculate the attention between highly correlated windows to achieve information exchange with other windows, thereby obtaining the global dependency of each window. At the same time, a feature block selection fusion mechanism is designed to effectively fuse the attention from three different perspectives, remove highly similar feature representations, and obtain a more robust feature representation. Based on the final feature representation that integrates local, global information and local correlation, a preliminary depth estimation result is generated, and the preliminary depth estimation result is input into the iterative optimization module to obtain the final monocular image depth estimation result.
[0087] The present invention solves the problem of missing information by obtaining global window information and interactive information between different windows. Through the feature block selection operation, the global and local features are effectively fused, avoiding a large number of similar features generated by the fusion attention mechanism, and the initial results are further corrected using the self-iteration module. Using the relationship between the depth map and the depth uncertainty map, a loss function including uncertainty loss and depth map loss is designed, and the initial generated results are iteratively optimized to achieve more accurate monocular image depth estimation. BRIEF DESCRIPTION OF THE DRAWINGS
[0088] Figure 1 is a monocular image depth estimation framework diagram of the method of the present invention in one embodiment;
[0089] Figure 2 1 is a schematic diagram of the principle of a global vector generator GVG in an embodiment of the method of the present invention;
[0090] Figure 3 It is a schematic diagram of a flow chart of generating an adjacency matrix representing affinity relationships between windows in an embodiment of the method of the present invention;
[0091] Figure 4 is a schematic diagram of a feature block selection and fusion FBSF module flow chart of a method of the present invention in one embodiment;
[0092] Figure 5 It is a schematic diagram of an uncertainty graph of the method of the present invention in one embodiment. DETAILED DESCRIPTION
[0093] It should be noted that:
[0094] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations on the technical solution of the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.
[0095] In the description of this application, "several" means more than one, "more" means more than two, "greater than", "less than", "exceed", etc. are understood to exclude the number itself, and "above", "below", "within", etc. are understood to include the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.
[0096] In the description of the present application, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0097] In this article, the character " / " generally indicates that the previous and next associated objects are in an "or" relationship.
[0098] Embodiment 1, as Figure 1 This is a framework diagram of a monocular image depth estimation method of the present invention. This framework diagram only shows the logical framework of the method described in this embodiment. In other possible embodiments of the present invention, different Figure 1 The illustrated framework performs the steps illustrated or described.
[0099] This embodiment provides a monocular image depth estimation method, such as Figure 1 As shown, including:
[0100] Obtain the monocular image to be estimated and preprocess it;
[0101] Input the preprocessed image into a pre-trained preliminary depth estimation network to obtain a preliminary depth estimation result of the image;
[0102] Inputting the preliminary depth estimation result into a pre-trained iterative optimization network to obtain a final monocular image depth estimation result;
[0103] The preliminary depth estimation network includes an encoder, a decoder and a regression head;
[0104] The encoder includes a plurality of sequentially connected encoding layers, and the decoder includes a plurality of sequentially connected decoding layers, wherein each encoding layer corresponds one-to-one with each decoding layer in the reverse direction of the decoding order along the encoding order; the decoding layer adopts a triplet cross attention TCA module,
[0105] The triplet cross attention TCA module includes a window attention WCA module, a cross-dimensional attention CDA module, a correlation attention REA module and a feature block selection fusion FBSF module;
[0106] The window attention WCA module is used to encode the input features and decode features Calculate attention and output window attention features ;
[0107] The cross-dimensional attention CDA module is used to decode the input features and the window attention feature Calculate attention and output cross-dimensional attention features ;
[0108] The correlation attention REA module is used to focus on the window attention feature Extract window correlation features and calculate attention, output correlation attention features ;
[0109] The feature block selects the fusion FBSF module to the window attention feature , the cross-dimensional attention feature and the correlation attention feature Fusion, generating the output features of the current decoding layer ;
[0110] The output feature of the last decoding layer is the final decoding feature; the encoding feature is the output feature of the encoding layer corresponding to the current decoding layer; if the current decoding layer is the first decoding layer, the decoding feature input to the current decoding layer is the output feature after the output features of each encoding layer are fused; if the current decoding layer is not the first decoding layer, the decoding feature input to the current decoding layer is the output feature of the previous decoding layer;
[0111] The regression head is used to generate the preliminary depth estimation result according to the final decoded feature.
[0112] like Figure 1 As shown, the preprocessed monocular image passes through the encoder and decoder to generate the final decoding features , the final decoded features Generate preliminary depth estimation results through regression head , preliminary depth estimation results After iterative optimization of the network, the final monocular image depth estimation result is generated.
[0113] The window in this embodiment 1 is consistent with the window size in swin-transformer, and each window size is 7*7.
[0114] In some embodiments, such as Figure 1 As shown, the window attention WCA module encodes the input features and decode features Computational attention includes:
[0115] S11: splitting the preprocessed image into windows;
[0116] S12: In each window, the coding feature Calculate the Q vector and K vector , for the decoding feature Calculate the V vector ,
[0117] Among them, the Q vector, K vector and V vector are respectively the Quary matrix, Key matrix and Value matrix obtained by linearly changing the corresponding features;
[0118] S13: calculated based on each window , and , calculate the window attention output as the window attention feature , the formula is:
[0119] ,
[0120] in, represents the window attention feature, represents computational attention, represents the normalization function, b is the relative position deviation matrix, and d represents and The ratio of the vector dimensions of b, each element in b represents the relative position embedding between position m and position n, where m represents The element position in n represents The position of the element in .
[0121] like Figure 1 As shown, the encoder based on Swin-Transformer contains 4 encoding layers, and the decoder contains 4 decoding layers, so Figure 1 The TCA module shown in , , , , , and G u The value range of parameter u is 1 / 2 / 3 / 4.
[0122] The decoding characteristics of each decoding layer TCA module The output feature generated by the previous layer of this layer, the encoding feature The output feature of the coding layer corresponding to the layer, the preliminary depth estimation network also includes a pyramid pooling PPM module, the pyramid pooling PPM module receives the output feature of each coding layer , the fusion process obtains the decoding feature V4 input into the first decoding layer.
[0123] The first decoding layer of the decoder corresponds to the last encoding layer of the encoder. In one embodiment, Figure 1 As shown, the encoding features and the decoding features of the first decoding layer of the decoder are and , the specific correspondence between the decoding features and encoding features of the TCA module in the decoding layer is: correspond , correspond , correspond and correspond .
[0124] like Figure 2 As shown, the cross-dimensional attention CDA module includes a global vector generator GVG, which is used to receive the decoded feature processing to generate a global perception vector , the process is:
[0125] S211: Decoding the feature The size of the dimension is compressed to be the same as the window dimension, and the sizes of the decoding feature dimension and the window dimension are respectively the decoding feature resolution and the window resolution;
[0126] S212: copying the compressed decoding feature to restore the original decoding feature dimension, and supplementing the edge with a mask;
[0127] S213: Extracting V vector from the compressed decoding features of each window size of the restored decoding features as the global perception vector .
[0128] The decoding characteristics Contains information about the entire image and decoding features of the TCA module at each decoding layer The dimension sizes are different, but the window size is 7*7, that is, the window resolution is 7*7.
[0129] like Figure 2 As shown in the figure, the features include channel dimension C, height dimension H and width dimension W. The channel dimension refers to the number of different "channels" or "feature maps" in the feature map; the height dimension refers to the height of the feature map, that is, the number of pixels in the feature map along the vertical direction; the width dimension refers to the width of the feature map, that is, the number of pixels in the feature map along the horizontal direction. Resolution refers to the number of pixels in the horizontal and vertical directions of the image. The window size described in the present invention refers to the window resolution. .
[0130] like Figure 1 As shown, the cross-dimensional attention CDA module decodes the input features and the window attention feature Computational attention includes:
[0131] S21: The global vector generator GVG generates the global perception vector , the global perception vector is used as the V vector , for the window attention feature Calculate the Q vector and K vector ;
[0132] S22: Q vector , K vector and V vector Calculate attention and output the cross-dimensional attention features , the formula is: ,
[0133] in, is the relative position deviation matrix, express and The ratio of the vector dimensions of Each element in represents the relative position embedding between position p and position q, where p represents The element position in , q represents The position of the element in .
[0134] Combination Figure 1 and Figure 3 , the correlation attention REA module pays attention to the window feature of the input Window correlation feature extraction and attention calculation include:
[0135] S31: The window attention feature Split into different windows, each window is linearly transformed to obtain the Q vector , K vector and V vector ;
[0136] S32: and Taking the average value in each window as the mark of the window, wherein taking the average value in each window means summing up the element values in the window and taking the average value;
[0137] S33: extract the and The flags of each window are generated accordingly and ;
[0138] S34: and Multiply the transposed matrix of to generate an adjacency matrix representing the affinity between windows ;
[0139] S35: constructing an index function based on the adjacency matrix to obtain which k windows each window has the highest correlation with, where k is a non-zero natural number less than the number of windows;
[0140] S36: respectively and Using the index function, a K vector containing window correlation information is generated through a collection function. and V vector ;
[0141] S37: Yes , and Calculate attention and output relevant attention features .
[0142] In the relevance attention REA module, only the k windows with the highest semantic relevance to the current window are taken, which promotes the information exchange and integration between different local ranges and reduces the computational complexity to a certain extent.
[0143] like Figure 3 As shown, step S31 sets the size to Window attention features Divided into different window areas, the window size is set to , the number of windows is N = , each window is linearly transformed to obtain the Q vector , K vector and V vector , , and The vector dimensions are , step S33 and The vector dimensions are Step 34 generates an adjacency matrix representing the affinity relationship between windows , as follows:
[0144] .
[0145] Step S35: Construct an index function based on the topk function in PyTorch based on the adjacency matrix , take the k token indexes with the highest correlation in the adjacency matrix, and get which k windows each window is most correlated with, expressed as , as follows:
[0146] ,
[0147] k is a non-zero natural number less than the number of windows, and can be set according to actual conditions. Generally, it does not exceed 1 / 4 of the number of windows. In one embodiment, k is set to 25.
[0148] Got it Then, step S35: collect the and The index result is the K vector containing the window correlation information. and V vector , as follows:
[0149] .
[0150] Step S36: , and Calculate attention and output relevant attention features , as follows:
[0151] .
[0152] like Figure 4 As shown, the feature block selects the fusion FBSF module to input the window attention feature , the cross-dimensional attention feature and the correlation attention feature Fusion includes:
[0153] S41: The cross-dimensional attention feature and the correlation attention feature The sum is set as feature G u ;
[0154] S42: The feature G u and the window attention feature Splice along the channel dimension to form feature B u ;
[0155] S43: The feature B u Split into feature blocks through slicing operation;
[0156] S44: The segmented feature blocks are selected by interval selection operation, and the maximum value in each selected feature block is extracted to generate feature B u ';
[0157] S45: For the feature B u 'Apply convolution operation to generate two channels, apply sigmoid function to each channel to generate a two-channel attention map, and convert the feature G u The window attention feature is multiplied with the dual-channel attention map to generate feature G u ' and feature L u ';
[0158] S46: The feature G u ' and feature L u 'Add the output features of the current decoding layer .
[0159] In step S44, "selecting the segmented feature blocks at intervals through a feature block selection operation" is to select feature blocks one by one at the same interval, and the interval is set to 8.
[0160] The initial depth estimate generated by TCA has significantly improved the effect of object edges and contours. However, in some images, such as Figure 5 The uncertainty diagram shown in the figure shows that for those cases where the colors are similar but the depths are different, the prediction of the depth change is wrong. In order to further improve the prediction accuracy, the depth estimation method of the present invention adds an iterative optimization network.
[0161] like Figure 1 As shown, the iterative optimization network includes an iterative optimizer based on a gated recurrent unit GRU, and the iterative optimization network obtains a final monocular image depth estimation result based on a preliminary depth estimation result, including:
[0162] S51: Computing Uncertain Graphs And the depth map of stage t The difference between the two represents the difference map at stage t , the formula is:
[0163] = ,
[0164] Among them, the initial depth map is the preliminary depth estimation result, the uncertainty map U is generated by the final decoded features through the regression head;
[0165] S52: The depth map of the tth stage , Uncertain Graph , the difference map of stage t , the hidden features of the tth stage and the initial hidden features Input gated recurrent unit GRU generates hidden features of the t+1th stage , the specific calculation is:
[0166] ,
[0167] in is the output feature of the first encoding layer, is the final decoding feature, such as Figure 1 As shown, for , for ;
[0168] S53: The hidden features of the t+1th stage The calculation result generated by the input update module △ is superimposed on the depth map of the tth stage , output the depth map of the t+1th stage , the specific calculation is:
[0169] = +△( ),
[0170] The update module △ is a Conv-ReLU combination operation. In a convolutional neural network, the convolution layer is responsible for extracting features from the input data, while the ReLU activation function performs nonlinear transformations on these features, enabling them to learn more complex patterns. This combination of Conv-ReLU is the basis for building deep convolutional neural networks, which allows the network to gradually learn more and more complex feature representations by stacking multiple such layers.
[0171] The iterative process of the iterative optimizer is as follows: set the number of iterations to x, where x is a non-zero natural number, and the initial value of t is 1, and sequentially execute steps S51-S53 in a loop until t is equal to x. is the final monocular image depth estimation result. In one embodiment, Figure 1 As shown, x is set to 3, t is initially 1, and steps S51-S53 are executed once to complete one iteration, and the output is ; t accumulates to 2, execute steps S51-S53 to complete the second iteration, and output ; t accumulates to 3, execute steps S51-S53 to complete the third iteration, and output , iterate 3 times until t is 3, is the final monocular image depth estimation result.
[0172] The training of the preliminary depth estimation network and the iterative optimization network includes a training loss function, and the adopted loss function includes an uncertainty loss and a depth map loss. The calculation of the uncertainty loss during the training process includes:
[0173] S61: Use the probability density function of the Laplace distribution to construct the uncertainty truth value of the i-th pixel :
[0174] ,
[0175] in The image depth truth value of the i-th pixel provided by the dataset, is the image depth prediction value of the i-th pixel, the image depth prediction is the depth map of each stage of the iterative optimization network, T represents the coefficient of controlling error tolerance, exp is an exponential function with e as the base; in one embodiment, T is set to 0.2;
[0176] S62: Calculating uncertainty loss : = ,
[0177] in, is the uncertainty value of the uncertainty map U of the i-th pixel, indicating the image uncertainty prediction value of the i-th pixel, It is the sum operation of i absolute values;
[0178] Calculating the depth map loss includes:
[0179] S71: Calculate the depth prediction value of the i-th pixel point of the image And the depth truth The difference in logarithmic functions between them is:
[0180] ;
[0181] S72: Calculate the depth map loss SILog as follows:
[0182] ,
[0183] Among them, i is not greater than y, y is the total number of pixels, is the variance minimization factor, is a scale constant; in this embodiment Set to 0.85, Set to 10;
[0184] Overall loss Calculate results for SILog Loss with uncertainty sum:
[0185] .
[0186] Excessive noise during training can disrupt the entire training process and force incorrect predictions. To mitigate the negative impact of depth noise, an uncertainty loss is introduced to correct the parts with high uncertainty, and a depth map loss is introduced to supervise the generated depth map to achieve more accurate depth estimation.
[0187] Pre-training a preliminary depth estimation network and iteratively optimizing the network includes the following:
[0188] Step 1, data preparation:
[0189] KITTI and NYU-v2 datasets are used to train and evaluate the network built by the present invention. The KITTI dataset is an outdoor image dataset including 23,488 training images and 697 test images, and the NYU-v2 dataset is an indoor image dataset involving 36,253 images for training and 654 images for testing.
[0190] Step 2, data preprocessing:
[0191] The input images of the KITTI dataset and NYU-v2 dataset are resized to 352×120 and 480×640 respectively, and random flipping, cropping, and rotation are used to augment the data.
[0192] Step 3, model building:
[0193] Build as Figure 1 The monocular image depth estimation model shown in the figure includes a preliminary depth estimation network and an iterative optimization network. The preliminary depth estimation network includes four encoding layers, four decoding layers and a regression head. Each decoding layer adopts a triplet cross attention TCA module. The iterative optimization network includes a GRU-based iterative optimizer.
[0194] Step 4, training process:
[0195] Design uncertainty loss Overlay Depth Map Loss The loss function training model is implemented based on the PyTorch deep learning framework. The model-related hyperparameters are set, the model training Epoch is set to 20, the Batch_size is set to 4, Adam is used as the optimizer, and the NVIDIA RTX 4090 GPU is used for acceleration. The learning rate is initialized to 2e-4 and linearly reduced to 2e-6. The models are trained in the training sets of the KITTI dataset and the NYU-v2 dataset respectively. After every 1000 training times, they are evaluated once in the test sets corresponding to each training set, and the model parameters corresponding to the best evaluation result are saved.
[0196] Step 5, training results:
[0197] The best saved parameters are applied to the preliminary depth estimation network and the iterative optimization network, thus completing the pre-training of the preliminary depth estimation network and the iterative optimization network. Using the pre-trained preliminary depth estimation network and the iterative optimization network can obtain more accurate monocular image depth estimation results.
[0198] The evaluation results of the NYU-v2 dataset and the KITTI dataset are shown in Table 1 (evaluation results on the NYU-v2 test dataset) and Table 2 (evaluation results on the KITTI test dataset), respectively. The last row of each table is the best evaluation result:
[0199] Table 1
[0200] ;
[0201] Table 2
[0202] ;
[0203] The present invention adopts average relative error (AbsRel), square relative difference (Sq rel), root mean square error (RMSE), average error ( ), threshold accuracy (δ1 / δ2 / δ3), the average relative error (AbsRel), square relative difference (Sq rel), root mean square error (RMSE), average error ( )The relationship between the size of these four indicators and the evaluation results is negatively correlated: the smaller the indicator, the better the evaluation result; the relationship between the size of the threshold accuracy (δ1 / δ2 / δ3) and the evaluation results is positively correlated: the larger the indicator, the better the evaluation result.
[0204] In Table 1 and Table 2, the first column of methods lists the names of various depth estimation methods, the second column of journals lists the names of the journals where the various methods are published, and the third to ninth columns are the above seven indicators, and each column of indicators lists the size of the indicator calculated by each method.
[0205] In Table 1 and Table 2, the results calculated by the method of the present invention show the average relative error (AbsRel), square relative difference (Sqrel), root mean square error (RMSE), average error ( ) are the smallest indicators, and the threshold accuracy (δ1 / δ2 / δ3) are the largest indicators.
[0206] In summary, the evaluation results of the method of the present invention on the NYU-v2 test dataset in Table 1 and the KITTI test dataset in Table 2 are the best.
[0207] The process of using the above pre-trained preliminary depth estimation network and iterative optimization network to perform monocular image depth estimation is as follows: Figure 1 As shown, including:
[0208] Obtain the monocular image to be estimated and preprocess it; input the preprocessed image into the pre-trained depth estimation network, and obtain the preliminary depth estimation result after passing through 4 encoding layers, 4 encoding layers and regression heads. ; The initial depth estimation result obtained Input the pre-trained iterative optimization network and output the final monocular image depth estimation result after 3 iterations. .
[0209] Embodiment 2, based on embodiment 1, this embodiment provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method described in embodiment 1 are implemented.
[0210] Embodiment 3, based on embodiment 1, this embodiment provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in embodiment 1 when executing the computer program.
[0211] Embodiment 4, based on embodiment 1, this embodiment provides a computer program product, including a computer program, which implements the steps of the method described in embodiment 1 when executed by a processor.
[0212] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0213] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0214] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0215] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0216] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A monocular image depth estimation method, characterized in that: include: Obtain the monocular image to be estimated and preprocess it; Input the preprocessed image into a pre-trained preliminary depth estimation network to obtain a preliminary depth estimation result of the image; Inputting the preliminary depth estimation result into a pre-trained iterative optimization network to obtain a final monocular image depth estimation result; The preliminary depth estimation network includes an encoder, a decoder and a regression head; The encoder includes a plurality of sequentially connected encoding layers, and the decoder includes a plurality of sequentially connected decoding layers, wherein each encoding layer corresponds one-to-one with each decoding layer in the reverse direction of the decoding order along the encoding order; the decoding layer adopts a triplet cross attention TCA module, The triplet cross attention TCA module includes a window attention WCA module, a cross-dimensional attention CDA module, a correlation attention REA module and a feature block selection fusion FBSF module; The window attention WCA module is used to calculate the attention of the input encoding features and decoding features, and output the window attention features; The cross-dimensional attention CDA module is used to calculate the attention of the input decoding features and the window attention features, and output the cross-dimensional attention features; The correlation attention REA module is used to extract the window correlation feature of the window attention feature and calculate the attention, and output the correlation attention feature; The feature block selection fusion FBSF module fuses the window attention feature, the cross-dimensional attention feature and the correlation attention feature to generate the output feature of the decoding layer; The output of the last decoding layer is the final decoding feature; The coding feature is the output feature of the coding layer corresponding to the current decoding layer; If the current decoding layer is the first decoding layer, the decoding features input to the current decoding layer are the output features of each encoding layer fused together; if the current decoding layer is not the first decoding layer, the decoding features input to the current decoding layer are the output features of the previous decoding layer; The regression head is used to generate the preliminary depth estimation result according to the final decoded feature.
2. The monocular image depth estimation method according to claim 1, characterized in that: The window attention WCA module calculates attention on the input encoding features and decoding features including: S11: splitting the preprocessed image into windows; S12: In each window, calculate the Q vector for the encoding feature and K vector , calculate the V vector for the decoded features , where Q vector, K vector and V vector are respectively the Quary matrix, Key matrix and Value matrix obtained by linearly changing the corresponding features; S13: Q vector calculated based on each window , K vector and V vector , calculate the window attention output as the window attention feature, the formula is: , in, represents the window attention feature, represents the normalization function, b is the relative position deviation matrix, and d represents the Q vector and K vector The ratio of the vector dimensions of b, each element in b represents the relative position embedding between position m and position n, where m represents the Q vector The element position in, n represents the K vector The position of the element in .
3. The monocular image depth estimation method according to claim 1, characterized in that: The preliminary depth estimation network also includes a pyramid pooling PPM module, which receives the output features of each encoding layer and performs fusion processing to obtain the decoding features input into the first decoding layer.
4. The monocular image depth estimation method according to claim 1, characterized in that: The cross-dimensional attention CDA module includes a global vector generator GVG, and the global vector generator GVG is used to receive the decoded feature processing to generate a global perception vector, and the process is: S211: compressing the size of the dimension of the decoding feature to be the same as the window dimension, wherein the size of the dimension of the decoding feature and the size of the window dimension are respectively the decoding feature resolution and the window resolution; S212: copying the compressed decoding feature to restore the dimension of the original decoding feature, and supplementing the edge with a mask; S213: Extracting a V vector from the compressed decoding features of each window size of the restored decoding features as a global perception vector.
5. The monocular image depth estimation method according to claim 4, characterized in that: The cross-dimensional attention CDA module is used to calculate the attention of the input decoding features and the window attention features, including: S21: The global vector generator GVG generates the global perception vector, the global perception vector is used as the V vector, and the window attention feature is calculated to obtain the Q vector and the K vector; S22: Calculate attention on the Q vector, K vector and V vector, and output the cross-dimensional attention feature.
6. The monocular image depth estimation method according to claim 1, characterized in that: The correlation attention REA module is used to extract the window correlation feature of the window attention feature and calculate the attention, including: S31: Split the window attention feature into different windows, and each window is linearly transformed to obtain a Q vector , K vector and V vector ; S32: Q vector and K vector Take the average value in each window as the mark of the window; S33: Extract the Q vectors respectively and K vector The flags of each window correspond to the generation of the intermediate Q vector matrix and the intermediate K vector matrix ; S34: The intermediate Q vector matrix and the intermediate K vector matrix Multiply the transposed matrix of to generate an adjacency matrix representing the affinity between windows; S35: constructing an index function based on the adjacency matrix to obtain which k windows each window has the highest correlation with, where k is a non-zero natural number less than the number of windows; S36: respectively performing the operations on the K vectors and V vector Use the index function to index and use the collection function to collect the corresponding index results to generate a K vector containing window correlation information and V vector containing window correlation information ; S37: Q vector , K vector containing window correlation information and V vector containing window correlation information Calculate attention and output relevant attention features.
7. The monocular image depth estimation method according to claim 1, characterized in that: The feature block selection fusion FBSF module fuses the window attention feature, the cross-dimensional attention feature and the correlation attention feature, including: S41: Set the sum of the cross-dimensional attention feature and the correlation attention feature as the first feature G u ; S42: The first feature G u The window attention feature is concatenated along the channel dimension to form the second feature B u ; S43: The second feature B u Split into feature blocks through slicing operation; S44: The segmented feature blocks are selected by interval selection operation, and the maximum value in each selected feature block is extracted to generate the third feature B u '; S45: For the third feature B u 'Apply convolution operation to generate two channels, apply sigmoid function to each channel to generate a two-channel attention map, and convert the first feature G u The window attention feature is multiplied with the dual-channel attention map to generate the fourth feature G u ' and the fifth characteristic L u '; S46: The fourth feature G u ' and the fifth characteristic L u 'Add together to generate the output features of this decoding layer.
8. The monocular image depth estimation method according to claim 1, characterized in that: The iterative optimization network includes an iterative optimizer based on a gated recurrent unit GRU, and the iterative optimization network obtains a final monocular image depth estimation result based on a preliminary depth estimation result, including: S51: Computing Uncertain Graphs And the depth map of stage t The difference between the two represents the difference map at stage t , the formula is: = , Among them, when t=1, is the initial depth map, whose value is the preliminary depth estimation result, and the uncertainty map Generated by the final decoded features through the regression head; S52: The depth map of the tth stage , Uncertain Graph , the difference map of stage t , the hidden features of the tth stage and the initial hidden features Input gated recurrent unit GRU generates hidden features of the t+1th stage , the specific calculation is: , in, is the output feature of the first encoding layer; S53: The hidden features of the t+1th stage The calculation result generated by the input update module △ is superimposed on the depth map of the tth stage , output the depth map of the t+1th stage , the specific calculation is: = +△( ), Among them, the update module △ is a Conv-ReLU combination operation, The iterative process of the iterative optimizer is as follows: set the number of iterations to x, where x is a non-zero natural number, and the initial value of t is 1, and sequentially execute steps S51-S53 in a loop until t is equal to x. is the final monocular image depth estimation result.
9. The monocular image depth estimation method according to claim 1, characterized in that: Training the preliminary depth estimation network and the iterative optimization network includes training a loss function, and the loss function used includes uncertainty loss and depth map loss; Calculating the uncertainty loss includes: S61: Use the probability density function of the Laplace distribution to construct the uncertainty truth value of the i-th pixel : , in, The image depth truth value of the i-th pixel provided by the dataset, is the image depth prediction value of the i-th pixel, the image depth prediction value is the depth map of each stage of the iterative optimization network, T represents the coefficient of control error tolerance, and exp is an exponential function with e as the base; S62: Calculating uncertainty loss : , in, is the uncertainty value of the uncertainty map of the i-th pixel, indicating the image uncertainty prediction value of the i-th pixel, It is the sum operation of i absolute values; Calculating the depth map loss includes: S71: Calculate the depth prediction value of the i-th pixel in the image And the depth truth The difference in logarithmic functions between them is: , S72: Calculate the depth map loss SILog as follows: , Where y is the total number of pixels, is the variance minimization factor, is the scale constant; Overall loss Calculate results for SILog Loss with uncertainty sum: 。 10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by a computing device, the computing device executes any one of the methods described in claims 1 to 9.
Citation Information
Patent Citations
Abnormity judgment method and device based on dynamic electrocardiogram and electronic equipment
CN116919419A
Monocular depth prediction method based on multilevel feature enhancement and cascade self-distillation
CN118261953A
Monocular depth estimation device and method and electronic equipment
CN118470079A
Dam defect time-sequence image description method based on local self-attention mechanism
WO2023217163A1
