A lightweight, fast and accurate self-supervised depth estimation method and system

Abstract

The application relates to the technical field of computer vision and image processing, in particular to a light, fast and accurate self-supervised depth estimation method and system, which can solve the problem of recovering the depth of a scene from a two-dimensional image and needing a complex depth estimation network model to a certain extent. The method comprises the following steps: acquiring an initial image data set; constructing a weighted coefficient matrix calculation module and a sparse coding module; based on the weighted coefficient matrix calculation module and the sparse coding module, a small depth estimation network model is constructed to speed up the algorithm reasoning speed and reduce the parameter quantity of the model; a large depth estimation network model is constructed and trained by using luminosity loss as a supervision signal; the small depth estimation network model is trained by using luminosity loss and function approximation loss as supervision signals to optimize the depth estimation precision of the small depth estimation network model; finally, input data is input into the small depth estimation network model to obtain an estimation result.

Description

Technical Field

[0001] This application relates to the fields of computer vision and image processing technology, and more specifically, to a lightweight, fast, and accurate self-supervised depth estimation method and system. Background Technology

[0002] With the development of computer vision and image processing technology, the ability to accurately recover the depth information of a scene from a two-dimensional image helps to better understand the three-dimensional structure of the scene and the distance relationship between the current camera and objects in the scene, thereby assisting in completing various visual tasks such as obstacle avoidance and endoscopic surgery.

[0003] However, ordinary cameras acquire two-dimensional images, losing the depth information of the scene. The depth of the scene can be recovered directly from monocular video in a completely unsupervised manner through the powerful data fitting ability of neural networks. Although the accuracy is greatly improved thanks to the powerful data fitting ability of neural networks.

[0004] However, complex depth estimation network models are often required to fit the data distribution characteristics in the input image. These complex models tend to increase the time and space complexity of the algorithm, slowing down the inference speed and increasing the number of parameters in the model. Summary of the Invention

[0005] To address the problem that existing technologies for recovering scene depth information from 2D images using neural networks require complex depth estimation network models, which slows down the algorithm's inference speed and increases the number of model parameters, this application provides a lightweight, fast, and accurate self-supervised depth estimation method and system.

[0006] The embodiments of this application are implemented as follows:

[0007] Firstly, this application provides a lightweight, fast, and accurate self-supervised depth estimation method, characterized by comprising:

[0008] An initial image dataset is obtained and preprocessed to obtain input data. The initial image dataset is a two-dimensional image dataset composed of monocular image sequences.

[0009] A weighted coefficient matrix calculation module is constructed to calculate the contribution of feature maps under different paths to the final output feature map. At the same time, based on the contribution, the original input feature map and the sparsely encoded feature map are fused at the pixel level to construct the sparse coding module.

[0010] Based on the weighted coefficient matrix calculation module and the sparse coding module, a small-scale depth estimation network model is constructed, and the input data is input into the small-scale depth estimation network model to obtain the estimation result.

[0011] In one possible implementation, the step of obtaining the initial image dataset and preprocessing the initial image dataset to obtain input data further includes the following steps:

[0012] The initial image dataset is subjected to random flipping, random cropping, and data normalization to obtain preliminary processed data;

[0013] The preliminary processed data is transformed to obtain the input data, which is a tensor data of size C×H×W, with the batch dimension omitted. Here, C represents the channel dimension of the sample, H represents the height of the input sample image, and W represents the width of the input sample image.

[0014] In one possible implementation, the module for constructing the weighted coefficient matrix calculation further includes the following steps:

[0015] Given the feature map of the i-th path:

[0016] Based on the channel dimension, the average value of the feature map under each path is calculated. And the average value of the feature map under each path By concatenating along the channel dimensions, a hybrid feature map is obtained.

[0017] According to formula X exp =δ hs (F exp (X fuse The hybrid feature map Mapping to a high-dimensional space yields a high-dimensional feature map.

[0018] Using formula X mid =δ hs (F mid (X exp For the high-dimensional feature map A nonlinear mapping process is performed to obtain an enhanced high-dimensional feature map.

[0019] According to formula X squ =F squ (X mid The enhanced high-dimensional feature map Mapping to the original feature space dimension The final output feature map is obtained.

[0020] The final output feature map is calculated based on the Softmax function. The probability values ​​of different channels but located at the same spatial position are used to construct a weighted coefficient matrix based on the probability values.

[0021] Where C is the number of channels in the feature map, H is the height of the feature map, W is the width of the feature map, N represents the number of paths, r is the dilation coefficient, and δ hs (·) represents the activation function, F exp (·),F mid (·),F squ (·) is the feature mapping function.

[0022] In one possible implementation, the step of fusing the original input feature map and the sparsely encoded feature map at the pixel level according to the degree of contribution, thereby constructing a sparse coding module, further includes the following steps:

[0023] Sparse modeling is performed in the high-dimensional channel space of the input feature map to obtain a sparse feature map.

[0024] The input feature map is then fused with the sparse feature map at the pixel level based on the contribution level calculated by the weighted coefficient matrix calculation module.

[0025] In one possible implementation, the step of performing sparse modeling in the high-dimensional channel space of the input feature map to obtain a sparse feature map further includes the following steps:

[0026] Given the input feature map According to the formula The input feature map is mapped to a high-dimensional channel feature space to obtain a high-dimensional input feature map.

[0027] Based on formula For the high-dimensional input feature map Each channel is encoded individually to obtain a high-dimensional sparse feature map.

[0028] In one possible implementation, the step of calculating the contribution level based on the weighted coefficient matrix calculation module and performing pixel-level fusion of the input feature map and the sparse feature map further includes the following steps:

[0029] Using formula The high-dimensional sparse feature map Compressed to the input feature map With the same feature space dimension, sparse feature maps are obtained.

[0030] According to the formula Calculate the weighted coefficient matrix of the feature maps under different paths, and based on the formula... The fused sparse feature map is calculated.

[0031] in, and This can be achieved using an economical and fast convolution kernel function with a kernel size of 1×1. This is implemented using a 3×3 convolution kernel function, instead of densely modeling the information between channels, where each channel feature is encoded separately.

[0032] In one possible implementation, the construction of a small-scale deep estimation network model based on the weighted coefficient matrix calculation module and the sparse coding module further includes the following steps:

[0033] The small-scale depth estimation network model adopts an encoder-decoder structure. Based on the resolution of the output feature map, the encoder and decoder are divided into 5 stages. Given the input feature map of the encoder at stage i ∈ {0,1,2,3,4},...

[0034] According to the formula The output feature map of the i-th stage encoder is calculated. Where F spblock (·) represents a sparse coding module, F down (·) is a convolution kernel function with a stride of 2 and a kernel size of 3×3;

[0035] According to the formula The output feature map of the i-th stage decoder is calculated.

[0036] Based on formula The parallax of the scene is obtained by regression.

[0037] In one possible implementation, after constructing a small-scale deep estimation network model based on the weighted coefficient matrix calculation module and the sparse coding module, the following steps are further included:

[0038] A large-scale depth estimation network model is constructed and trained using photometric loss as a supervision signal to obtain a high-precision large-scale depth estimation network model, and the model weights of the high-precision large-scale depth estimation network model are frozen.

[0039] Photometric loss and function approximation loss are used as supervision signals to train the small depth estimation network model, which in turn simulates the data distribution characteristics of the feature maps at each stage of the high-precision depth estimation model, thereby optimizing the depth estimation accuracy of the small depth estimation network model.

[0040] Secondly, this application provides a lightweight, fast, and accurate self-supervised depth estimation system, comprising:

[0041] The input module is used to acquire an initial image dataset and preprocess the initial image dataset to obtain input data. The initial image dataset is a two-dimensional image dataset composed of monocular image sequences.

[0042] The calculation module is used to construct the weighted coefficient matrix calculation module, which is used to calculate the contribution of the feature map under different paths to the final output feature map. At the same time, according to the contribution, the original input feature map and the sparsely encoded feature map are fused at the pixel level to construct the sparse coding module.

[0043] The processing module is used to construct a small depth estimation network model based on the weighted coefficient matrix calculation module and the sparse coding module, and input the input data into the small depth estimation network model to obtain the estimation result.

[0044] In one possible implementation, the lightweight depth estimation system further includes:

[0045] The calibration module is used to construct and train a large-scale depth estimation network model with photometric loss as a supervision signal to obtain a high-precision depth estimation model and freeze the model weights of the high-precision depth estimation model.

[0046] An optimization module is used to train the small depth estimation network model using photometric loss and function approximation loss as supervision signals, driving the small depth estimation network model to simulate the data distribution characteristics of the feature maps at each stage of the high-precision depth estimation model, so as to optimize the depth estimation accuracy of the small depth estimation network model.

[0047] The technical solution provided in this application can achieve at least the following beneficial effects:

[0048] This application provides a lightweight, fast, and accurate self-supervised depth estimation method and system. The method and system calculate the contribution of feature maps from different paths to the final output feature map using a weighted coefficient matrix calculation module, thereby driving more efficient pixel-level fusion of feature maps from different paths. A sparse feature encoding module accelerates the model's inference speed and reduces the number of model parameters. Function approximation loss drives the features of each stage of the lightweight, fast, small-scale depth estimation model to approximate the distribution characteristics of feature maps at each stage of a complex and time-consuming large-scale depth estimation network model, thereby improving the depth estimation accuracy of the small model. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart illustrating a lightweight, fast, and accurate self-supervised depth estimation method according to an exemplary embodiment of this application;

[0051] Figure 2 This is a schematic diagram illustrating the process of obtaining input data, as shown in another exemplary embodiment of this application;

[0052] Figure 3 This is a flowchart illustrating the construction of a weighted coefficient matrix calculation module and a sparse coding module, as shown in another exemplary embodiment of this application.

[0053] Figure 4 This is a schematic diagram illustrating the sparse modeling process in another exemplary embodiment of this application;

[0054] Figure 5 This is a schematic diagram illustrating the process of fusing an input feature map and a sparse feature map, as shown in another exemplary embodiment of this application.

[0055] Figure 6 This is a schematic diagram illustrating the process of constructing a small depth estimation network model, as shown in another exemplary embodiment of this application;

[0056] Figure 7 This is a schematic diagram illustrating the process of training a lightweight depth estimation model by introducing a functional approximation loss as an additional supervision signal, as shown in another exemplary embodiment of this application.

[0057] Figure 8 This is a schematic diagram illustrating the structure of a lightweight, fast, and accurate self-supervised depth estimation system according to an exemplary embodiment of this application;

[0058] Figure 9 This is a schematic diagram of a weighted coefficient matrix calculation module shown in an exemplary embodiment of this application;

[0059] Figure 10 This is a schematic diagram of a sparse feature encoding module shown in an exemplary embodiment of this application;

[0060] Figure 11 This is a schematic diagram of a small depth estimation network model connection shown in an exemplary embodiment of this application;

[0061] Figure 12This is a schematic block diagram illustrating the overall training of a small depth estimation network model as shown in an exemplary embodiment of this application.

[0062] Figure label:

[0063] 700 Lightweight depth estimation system; 710 Input module; 720 Calculation module; 730 Processing module; 731 Calibration module; 732 Optimization module. Detailed Implementation

[0064] To make the objectives, implementation methods and advantages of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0065] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0066] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0067] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0068] Before explaining the lightweight, fast, and accurate self-supervised depth estimation method and system provided in the embodiments of this application, the application scenarios and implementation environment of the embodiments of this application will be introduced first.

[0069] With the development of computer vision and image processing technology, the ability to accurately recover the depth information of a scene from a two-dimensional image helps to better understand the three-dimensional structure of the scene and the distance relationship between the current camera and objects in the scene, thereby assisting in completing various visual tasks such as obstacle avoidance and endoscopic surgery.

[0070] However, ordinary cameras acquire two-dimensional images, losing the depth information of the scene. The depth of the scene can be recovered directly from monocular video in a completely unsupervised manner through the powerful data fitting ability of neural networks. Although the accuracy is greatly improved thanks to the powerful data fitting ability of neural networks.

[0071] However, complex depth estimation network models are often required to fit the data distribution characteristics in the input image. These complex models tend to increase the time and space complexity of the algorithm, slowing down the inference speed and increasing the number of parameters in the model.

[0072] Based on this, this application provides a lightweight, fast, and accurate self-supervised depth estimation method and system. It calculates the contribution of feature maps from different paths to the final output feature map by simultaneously modeling the contextual dependencies of pixels in feature maps along the same path and the dependencies between feature maps from different paths, using a weighted coefficient matrix calculation module. Pixel-level fusion of feature maps obtained from different paths is then achieved based on this contribution. A sparse coding module is used to sparsely model channel information, accelerating the model's inference speed and reducing its parameters. A small-scale depth estimation network model is constructed based on the weighted coefficient matrix calculation module and the sparse coding module. Functional approximation loss is used to drive the feature maps output at each stage of the small-scale depth estimation network model to approximate the distribution characteristics of the corresponding stage output feature maps of the large-scale depth estimation model, thereby improving the accuracy of the scene depth estimated by the small-scale depth estimation network model.

[0073] Next, the technical solutions of this application and how they solve the aforementioned technical problems will be described in detail through embodiments and in conjunction with the accompanying drawings. The embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this application.

[0074] Figure 1 This is a flowchart illustrating a lightweight, fast, and accurate self-supervised depth estimation method according to an exemplary embodiment of this application.

[0075] In one exemplary embodiment, such as Figure 1 As shown, a lightweight, fast, and accurate self-supervised depth estimation method is provided. In this embodiment, the method may include the following steps:

[0076] Step 100: Obtain the initial image dataset and preprocess the initial image dataset to obtain input data. The initial image dataset is a two-dimensional image dataset composed of monocular image sequences.

[0077] Step 200: Construct a weighted coefficient matrix calculation module to calculate the contribution of feature maps under different paths to the final output feature map. At the same time, based on the contribution, perform pixel-level fusion of the original input feature map and the sparsely encoded feature map to construct a sparse coding module.

[0078] Step 300: Based on the weighted coefficient matrix calculation module and the sparse coding module, construct a small depth estimation network model, and input the input data into the small depth estimation network model to obtain the estimation result.

[0079] As can be seen, this embodiment uses a weighted coefficient matrix calculation module to calculate the contribution of feature maps under different paths to the final output feature map, thereby driving feature maps of different paths to achieve more efficient pixel-level fusion; and uses a sparse feature encoding module to accelerate the inference speed of the model and reduce the number of model parameters.

[0080] Figure 2 This is a schematic diagram illustrating the process of obtaining input data, as shown in another exemplary embodiment of this application.

[0081] In one possible implementation, such as Figure 2 As shown, the process of obtaining the initial image dataset and preprocessing the initial image dataset to obtain input data further includes the following steps:

[0082] Step 110: Perform random flipping, random cropping, and data normalization on the initial image dataset to obtain preliminary processed data;

[0083] Step 120: Transform the preliminary processed data to obtain the input data, which is a tensor data of size C×H×W, with the batch dimension omitted. Here, C represents the channel dimension of the sample, H represents the height of the input sample image, and W represents the width of the input sample image.

[0084] The input data can be used as input to the depth estimation network and the camera pose network. C represents the channel dimension of the sample. In the depth estimation network, C=3, and in the camera pose network, C=9. H represents the height of the input sample image. In this embodiment, H=128. W represents the width of the input sample image. W=416.

[0085] Figure 3 This is a flowchart illustrating the construction of the weighted coefficient matrix calculation module and the sparse coding module, as shown in another exemplary embodiment of this application. Figure 9 This is a schematic diagram of a weighted coefficient matrix calculation module shown in an exemplary embodiment of this application.

[0086] In one possible implementation, such as Figure 3As shown, the module for constructing the weighted coefficient matrix calculation also includes the following steps:

[0087] Step 210: Given the feature map of the i-th path as follows

[0088] Step 220: Calculate the average value of the feature map for each path based on the channel dimension. And the average value of the feature map under each path By concatenating along the channel dimensions, a hybrid feature map is obtained.

[0089] Step 230: According to formula X exp =δ hs (F exp (X fuse )), the hybrid features Figure X fuse Mapping to a high-dimensional space yields a high-dimensional feature map.

[0090] Step 240: Using formula X mid =δ hs (F mid (X exp The high-dimensional feature map is then subjected to nonlinear mapping to obtain an enhanced high-dimensional feature map.

[0091] Step 250: According to formula X squ =F squ (X mid The enhanced high-dimensional feature map is mapped to the original feature space dimension. The final output feature map is obtained.

[0092] Step 260: Calculate the final output feature map based on the Softmax function. The probability values ​​of different channels but located at the same spatial position are used to construct a weighted coefficient matrix based on the probability values.

[0093] In this process, to model the weighted coefficient matrix of feature maps across different paths, the mixed feature map is mapped to a high dimension. To enhance the expressive power of the high-dimensional features, the high-dimensional feature map is then subjected to a nonlinear mapping. C represents the number of channels in the feature map, H represents the height of the feature map, W represents the width of the feature map, N represents the number of paths, r is the dilation coefficient, and δ... hs (·) represents the activation function, F exp (·),F mid (·),F squ (·) represents the feature mapping function. In this embodiment, r = 2, and the feature mapping function F...exp (·),F mid (·),F squ (·) can be implemented using a convolution kernel with a kernel size of 3×3 and a stride of 1.

[0094] As can be seen, in order to better fuse feature maps obtained from different paths at the pixel level, this embodiment constructs, as follows: Figure 9 The weighted coefficient matrix calculation module shown calculates the contribution of feature maps on different paths to the final output feature map by simultaneously modeling the contextual dependencies of pixels in feature maps on the same path and the dependencies between feature maps on different paths. Finally, it performs pixel-level weighted fusion of feature maps on different paths based on the calculated pixel-level weights.

[0095] Figure 10 This is a schematic diagram of a sparse feature encoding module shown in an exemplary embodiment of this application.

[0096] In one possible implementation, such as Figure 3 As shown, the step of fusing the original input feature map and the sparsely encoded feature map at the pixel level according to the contribution level to construct the sparse coding module further includes the following steps:

[0097] Step 270: Perform sparse modeling in the high-dimensional channel space of the input feature map to obtain a sparse feature map;

[0098] Step 280: Based on the contribution level calculated by the weighted coefficient matrix calculation module, the input feature map and the sparse feature map are fused at the pixel level.

[0099] As can be seen, in order to accelerate the inference speed of the model while maintaining its inference accuracy, we constructed the following... Figure 10 The sparse feature encoding module F shown spblock (·), This module accelerates model inference by sparsely modeling channel information.

[0100] Figure 4 This is a schematic diagram illustrating the process of sparse modeling, as shown in another exemplary embodiment of this application.

[0101] In one possible implementation, such as Figure 4 As shown, the step of performing sparse modeling in the high-dimensional channel space of the input feature map to obtain a sparse feature map further includes the following steps:

[0102] Step 271: Given the input feature map as According to the formula The input feature map is mapped to a high-dimensional channel feature space to obtain a high-dimensional input feature map.

[0103] Step 272: Based on the formula For the high-dimensional input feature map Each channel is encoded individually to obtain a high-dimensional sparse feature map.

[0104] It can be seen that, in order to compensate for the inability of channels to interact due to sparse modeling of channel information, low-dimensional channel features are mapped to a high-dimensional channel feature space. To enable information exchange with the original input feature map containing information relationships between channels, the high-dimensional input feature map... Each channel is encoded individually.

[0105] Figure 5 This is a schematic diagram illustrating the process of fusing an input feature map with a sparse feature map, as shown in another exemplary embodiment of this application.

[0106] In one possible implementation, such as Figure 5 As shown, the step of calculating the contribution level based on the weighted coefficient matrix calculation module and fusing the input feature map with the sparse feature map at the pixel level further includes the following steps:

[0107] Step 281: Using the formula The high-dimensional sparse feature map Compressed to the input feature map With the same feature space dimension, sparse feature maps are obtained.

[0108] Step 282: According to the formula Calculate the weighting coefficient matrix of feature maps for different paths. And based on the formula The fused sparse feature map is calculated.

[0109] in, and This can be achieved using an economical and fast convolution kernel function with a kernel size of 1×1. This is implemented using a 3×3 convolution kernel function, instead of densely modeling the information between channels, where each channel feature is encoded separately.

[0110] As can be seen, this embodiment groups the feature maps to be processed, uses dense modeling for the feature maps within a group, and cuts off the information interaction between groups to speed up inference time.

[0111] Figure 6 This is a schematic diagram illustrating the process of constructing a small depth estimation network model, as shown in another exemplary embodiment of this application. Figure 11This is a schematic diagram of a small depth estimation network model connection shown in an exemplary embodiment of this application.

[0112] In one possible implementation, such as Figure 6 As shown, the construction of a small-scale deep estimation network model based on the weighted coefficient matrix calculation module and the sparse coding module further includes the following steps:

[0113] Step 310: The small-scale depth estimation network model adopts an encoder-decoder structure. Based on the resolution of the output feature map, the encoder and decoder are divided into 5 stages. Given the input feature map of the encoder at stage i ∈ {0,1,2,3,4},...

[0114] Step 320: According to the formula The output feature map of the i-th stage encoder is calculated.

[0115] Step 330: According to the formula The output feature map of the i-th stage decoder is calculated.

[0116] Step 340: Based on the formula The parallax of the scene is obtained by regression.

[0117] Among them, F spblock (·) represents a sparse coding module, F down (·) is a convolution kernel function with a stride of 2 and a kernel size of 3×3, F disp (·) It consists of a 3×3 convolutional kernel function with a stride of 1 and a sigmoid activation function, connected as follows: Figure 11 As shown.

[0118] Figure 7 This is a schematic diagram illustrating the process of training a lightweight depth estimation model by introducing a functional approximation loss as an additional supervision signal, as shown in another exemplary embodiment of this application. Figure 12 This is a schematic block diagram illustrating the overall training of a small depth estimation network model as shown in an exemplary embodiment of this application.

[0119] In one possible implementation, such as Figure 7 As shown, after constructing a small-scale deep estimation network model based on the weighted coefficient matrix calculation module and the sparse coding module, the following steps are further included:

[0120] Step 400: Construct and train a large-scale depth estimation network model using photometric loss as a supervision signal to obtain a high-precision large-scale depth estimation network model, and freeze the model weights of the high-precision large-scale depth estimation network model.

[0121] In some embodiments of this application, the depth estimation network invented in the invention patent CN202111488926.6 "Deep Estimation Method, Device, Electronic Device and Storage Medium of Image" by Cheng Jun et al. is first used as a large-scale depth estimation network model. The large-scale depth estimation network model, bridging network and camera pose network are trained in a fully unsupervised manner to obtain a high-precision large-scale depth estimation model. Then the model weights of the trained large-scale depth estimation network model and bridging network are frozen.

[0122] Step 500: Using photometric loss and function approximation loss as supervision signals, train the small depth estimation network model to simulate the data distribution characteristics of the feature maps at each stage of the high-precision depth estimation model, so as to optimize the depth estimation accuracy of the small depth estimation network model.

[0123] Specifically, the output feature maps of the i-th stage encoder and decoder of the small depth estimation network model are respectively The disparity at the i-th scale is

[0124] The output feature maps of the encoder and decoder in the i-th stage of the large-scale depth estimation network model are respectively The disparity at the i-th scale is

[0125] To further reduce the number of channels in each stage and align the channel counts of corresponding stages, enabling the use of lighter and faster small-scale depth estimation network models to simulate large-scale depth estimation network models, this scheme utilizes a pre-trained bridging network to compress the feature maps output by the encoder and decoder at each stage of the large-scale depth estimation network model. This allows the output of each stage of the small-scale depth estimation network model to approximate the more compact feature maps.

[0126] in, It is the number of channels in the output feature map of the encoder / decoder in the i-th stage of a small depth estimation network model.

[0127] Will It serves as the input to the encoder stage (i+1) of a large-scale depth estimation network model / the input to the decoder stage (i-1).

[0128] in, It consists of a convolution kernel function with a kernel size of 3×3 and a stride of 1, a Hardswish activation function, and an information fine-tuning module from Cheng Jun et al.'s invention patent CN202111488926.6 "Image Depth Estimation Method, Device, Electronic Equipment and Storage Medium", which are used for feature map compression and information fine-tuning, respectively. It consists of a 3×3 convolutional kernel function with a stride of 1 and a Hardswish activation function, and is used for feature map channel dimension recovery.

[0129] Where, λ enc ,λ dec ,λ disp This corresponds to the loss weighting hyperparameter, which is set to λ in this scheme. enc =0.05,λ dec =0.1,λ disp =10.

[0130] To further improve the accuracy of small-scale depth estimation network models, this application uses photometric loss and function approximation loss as additional supervision signals. It uses the image reconstruction objective function invented in Wang Fei et al.'s invention patent CN202110346713.3, "Method, Apparatus, Terminal Equipment and Storage Medium for Estimating Image Scene Depth," as the photometric loss in this technical solution. Simultaneously, based on:

[0131]

[0132] As a function approximation loss.

[0133] Step 600: Input the input data into the small depth estimation network model to obtain the estimation result.

[0134] Among them, the optimized small depth estimation network model can estimate the corresponding scene depth from monocular images, and can also estimate the relative pose between adjacent frames using camera pose alone.

[0135] It can be seen that, in order to further improve the estimation accuracy of small-scale deep estimation network models, such as Figure 12 As shown, this embodiment uses photometric loss and function approximation loss to drive the features of each stage of the lightweight and fast small depth estimation model to approximate the distribution characteristics of the feature maps of each stage of the complex and time-consuming large depth estimation network model, thereby improving the depth estimation accuracy of the small model.

[0136] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially as indicated, these steps are not necessarily executed in the indicated order. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0137] Corresponding to the aforementioned embodiment of a lightweight, fast, and accurate self-supervised depth estimation method, this application also provides an embodiment of a lightweight, fast, and accurate self-supervised depth estimation system using the same technical concept.

[0138] Figure 8 This is a schematic diagram illustrating the structure of a lightweight, fast, and accurate self-supervised depth estimation system according to an exemplary embodiment of this application.

[0139] In one exemplary embodiment, such as Figure 8 As shown, the depth estimation system 700 includes:

[0140] The input module 710 is used to acquire an initial image dataset and preprocess the initial image dataset to obtain input data. The initial image dataset is a two-dimensional image dataset composed of monocular image sequences.

[0141] The calculation module 720 is used to construct a weighted coefficient matrix calculation module, which is used to calculate the contribution of feature maps under different paths to the final output feature map. At the same time, based on the contribution, the original input feature map and the sparsely encoded feature map are fused at the pixel level to construct a sparse coding module.

[0142] The processing module 730 is used to construct a small depth estimation network model based on the weighted coefficient matrix calculation module and the sparse coding module, and input the input data into the small depth estimation network model to obtain the estimation result.

[0143] In one possible implementation, such as Figure 8 As shown, the lightweight depth estimation system further includes:

[0144] The calibration module 731 is used to construct and train a large-scale depth estimation network model with photometric loss as a supervision signal to obtain a high-precision depth estimation model and freeze the model weights of the high-precision depth estimation model.

[0145] The optimization module 732 is used to train the small depth estimation network model using photometric loss and function approximation loss as supervision signals, thereby driving the small depth estimation network model to simulate the data distribution characteristics of the feature maps at each stage of the high-precision depth estimation model, so as to optimize the depth estimation accuracy of the small depth estimation network model.

[0146] For specific limitations regarding lightweight depth estimation systems, please refer to the limitations of lightweight, fast, and accurate self-supervised depth estimation methods described above, which will not be repeated here. Each module in the aforementioned lightweight depth estimation system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0147] It should be understood that the technical solutions in the embodiments of this application can be implemented using software plus the necessary general-purpose hardware platform. Therefore, the technical solutions in the embodiments of this application, in essence or in part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a computer-readable storage medium.

[0148] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0149] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A lightweight, fast, and accurate self-supervised depth estimation method, characterized in that, include: An initial image dataset is obtained and preprocessed to obtain input data. The initial image dataset is a two-dimensional image dataset composed of monocular image sequences. A weighted coefficient matrix calculation module is constructed to calculate the contribution of feature maps under different paths to the final output feature map. At the same time, based on the contribution, the original input feature map and the sparsely encoded feature map are fused at the pixel level to construct the sparse coding module. Based on the weighted coefficient matrix calculation module and the sparse coding module, a small-scale depth estimation network model is constructed, and the input data is input into the small-scale depth estimation network model to obtain the estimation result. The step of fusing the original input feature map and the sparsely encoded feature map at the pixel level according to the contribution level to construct the sparse coding module also includes the following steps: Sparse modeling is performed in the high-dimensional channel space of the input feature map to obtain a sparse feature map. The input feature map is then fused with the sparse feature map at the pixel level based on the contribution level calculated by the weighted coefficient matrix calculation module.

2. The lightweight, fast, and accurate self-supervised depth estimation method as described in claim 1, characterized in that, The process of obtaining the initial image dataset and preprocessing the initial image dataset to obtain input data further includes the following steps: The initial image dataset is subjected to random flipping, random cropping, and data normalization to obtain preliminary processed data; The preliminary processed data is transformed to obtain the input data, which is: The input data is a tensor data set, where the batch dimension is omitted. Here, C represents the channel dimension of the sample, H represents the height of the input sample image, and W represents the width of the input sample image.

3. The lightweight, fast, and accurate self-supervised depth estimation method as described in claim 1, characterized in that, The module for constructing the weighted coefficient matrix calculation also includes the following steps: Given the feature map of the i-th path: ; N The number of paths; Based on the channel dimension, the average value of the feature map under each path is calculated. And the average value of the feature map under each path. By concatenating along the channel dimensions, a hybrid feature map is obtained. ; According to the formula The hybrid feature map Mapping to a high-dimensional space yields a high-dimensional feature map. ; Using formula For the high-dimensional feature map A nonlinear mapping process is performed to obtain an enhanced high-dimensional feature map. ; According to the formula The enhanced high-dimensional feature map Mapping to the original feature space dimension The final output feature map is obtained. ; The final output feature map is calculated based on the Softmax function. The probability values ​​of different channels but located at the same spatial position are used to construct a weighted coefficient matrix based on the probability values. Where C is the number of channels in the feature map, H is the height of the feature map, W is the width of the feature map, N represents the number of paths, and r is the dilation coefficient. This represents the activation function. This is the feature mapping function.

4. The lightweight, fast, and accurate self-supervised depth estimation method as described in claim 3, characterized in that, The step of performing sparse modeling in the high-dimensional channel space of the input feature map to obtain a sparse feature map further includes the following steps: Given the input feature map According to the formula Mapping a given input feature map to a high-dimensional channel feature space yields a high-dimensional input feature map. ; Given an input feature map; C is the number of channels in the corresponding feature map; H is the height of the corresponding feature map; W is the width of the corresponding feature map; A function that maps a given input feature map to a high-dimensional feature space; Based on formula For the high-dimensional input feature map Each channel is encoded individually to obtain a high-dimensional sparse feature map. ; For high-dimensional feature maps The function that encodes each channel individually is implemented in this implementation by a 3×3 convolution kernel function that disconnects the channels; r is the coefficient of thermal expansion.

5. The lightweight, fast, and accurate self-supervised depth estimation method as described in claim 4, characterized in that, The step of calculating the contribution level based on the weighted coefficient matrix calculation module and performing pixel-level fusion of the input feature map and the sparse feature map further includes the following steps: Using formula The high-dimensional sparse feature map Compress to a given input feature map With the same feature space dimension, sparse feature maps are obtained. ; According to the formula The weighted coefficient matrix of feature maps for different paths is calculated, and based on the formula... The sparse feature map after fusion is calculated. ; They respectively represent the calculated corresponding to The weighted coefficient matrix; This is the module for calculating the weighted coefficient matrix; W1 is... W2 is ; in, and Using a kernel size of The implementation of the convolution kernel function, Based on kernel size Instead of densely modeling the information between channels, the convolution kernel function is used to encode the features of each channel separately.

6. The lightweight, fast, and accurate self-supervised depth estimation method as described in claim 1, characterized in that, The construction of a small-scale deep estimation network model based on the weighted coefficient matrix calculation module and the sparse coding module further includes the following steps: The small-scale depth estimation network model adopts an encoder-decoder structure, dividing the encoder and decoder into 5 stages based on the resolution of the output feature map. Given the encoder's first stage... The input feature map for the stage is ; According to the formula The output feature map of the encoder in the i-th stage is calculated. ,in For sparse coding modules, It has a step size of 2 and a kernel size of The convolution kernel function; According to the formula The output feature map of the i-th stage decoder is calculated. ; Based on formula This yields the parallax of the scene; in, The input feature maps are respectively the encoder of the small depth estimation network model in the i-th stage. The number of channels, the encoder output feature map of the small depth estimation network model in the i-th stage The number of channels; , Let F be the input feature map of the i-th stage decoder and the output feature map of the i-th stage decoder, respectively; up F is an upsampling function consisting of a convolution kernel function with a stride of 1 and a kernel size of 3, and a bidirectional interpolation function; ct It is a channel transform function composed of convolution kernel functions with a stride of 1 and a kernel size of 3; The corresponding value calculated by the weighted coefficient calculation module is... The weighted coefficient matrix of the calculated feature map; j is the scale index regressed by the small-scale deep estimation network model.

7. The lightweight, fast, and accurate self-supervised depth estimation method as described in claim 1, characterized in that, After constructing a small-scale deep estimation network model based on the weighted coefficient matrix calculation module and the sparse coding module, the following steps are further included: A large-scale depth estimation network model is constructed and trained using photometric loss as a supervision signal to obtain a high-precision large-scale depth estimation network model, and the model weights of the high-precision large-scale depth estimation network model are frozen. Photometric loss and function approximation loss are used as supervision signals to train the small depth estimation network model, which in turn drives the small depth estimation network model to simulate the data distribution characteristics of the feature maps at each stage of the high-precision depth estimation model, thereby optimizing the depth estimation accuracy of the small depth estimation network model.

8. A lightweight, fast, and accurate self-supervised depth estimation system, wherein the system is used to implement the lightweight, fast, and accurate self-supervised depth estimation method as described in any one of claims 1 to 7, characterized in that, The system includes: The input module is used to acquire an initial image dataset and preprocess the initial image dataset to obtain input data. The initial image dataset is a two-dimensional image dataset composed of monocular image sequences. The calculation module is used to construct the weighted coefficient matrix calculation module, which is used to calculate the contribution of the feature map under different paths to the final output feature map. At the same time, according to the contribution, the original input feature map and the sparsely encoded feature map are fused at the pixel level to construct the sparse coding module. The processing module is used to construct a small depth estimation network model based on the weighted coefficient matrix calculation module and the sparse coding module, and input the input data into the small depth estimation network model to obtain the estimation result.

9. The lightweight, fast, and accurate self-supervised depth estimation system as described in claim 8, characterized in that, Depth estimation systems also include: The calibration module is used to construct and train a large-scale depth estimation network model with photometric loss as a supervision signal to obtain a high-precision depth estimation model and freeze the model weights of the high-precision depth estimation model. An optimization module is used to train the small depth estimation network model using photometric loss and function approximation loss as supervision signals, driving the small depth estimation network model to simulate the data distribution characteristics of the feature maps at each stage of the high-precision depth estimation model, so as to optimize the depth estimation accuracy of the small depth estimation network model.

Images