Depth-of-field estimator based on monocular RGB camera

Abstract

The invention discloses a depth-of-field estimator based on a monocular RGB camera, and belongs to the technical field of depth-of-field estimation of monocular RGB cameras, and the depth-of-field estimator comprises a sensor, a depth-of-field calculation unit and an output unit. The sensor collects a single RGB image; the depth-of-field calculation unit generates a depth map through a causal effect among modeling variables based on a spatial probability logic neural network of a variational automatic encoder; and the output unit analyzes the depth map into a downstream task signal. According to the invention, the variational inference framework is adopted to construct the model, the computing power demand is greatly reduced, the minimum model parameter quantity is only 10.3 M, the deployment memory is as low as 30 MB or below, the operation of low-cost embedded equipment is supported, the fastest single-frame inference duration is 1 ms, and the maximum depth-of-field estimation distance can reach 1000 m. The method effectively solves the problems of sparse sampling of a traditional laser radar and myopia of a depth camera, and is suitable for space sensing modules of maneuvering carriers such as a multi-rotor unmanned aerial vehicle and a wheeled vehicle.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and deep learning technology, specifically, it relates to a depth estimator based on a monocular RGB camera. Background Technology

[0002] In the fields of computer vision and robotics, spatial perception technology is core to enabling intelligent devices to understand and interact with their environment. Traditional spatial perception methods mainly rely on hardware devices such as LiDAR or depth cameras. LiDAR generates 3D spatial information through point-to-point ranging and sampling, but the sparsity of its sampling points limits high-precision reconstruction of continuous 3D space. Depth cameras use the parallax principle for depth estimation, but due to the nonlinear characteristics of parallax error, their effective measurement range is usually limited to within 15 meters, making it difficult to meet the needs of long-distance scenes. In addition, these hardware devices are usually expensive and have poor adaptability to vehicle or airborne environments, limiting their widespread application in low-cost embedded devices.

[0003] In recent years, depth estimation techniques based on monocular RGB images have gradually become a research hotspot. This technique extracts features from a single-view RGB image and infers depth information, avoiding the limitations of traditional hardware devices. However, existing deep learning-based monocular depth estimation models typically require substantial computational resources for training and inference, resulting in a large number of model parameters and leading to high hardware requirements and long inference times in practical deployments. Furthermore, existing models perform poorly in handling ultra-long-distance depth estimation, making it difficult to meet the needs of ultra-long-distance continuous 3D spatial reconstruction in application scenarios such as drones and autonomous vehicles.

[0004] Therefore, there is an urgent need for a monocular RGB spatial perception technology that can operate efficiently in low-computing-power environments and support ultra-long-distance depth estimation. This technology not only needs to address the high dependence of existing models on computing resources, but also needs to overcome the shortcomings of traditional hardware devices in spatial perception, such as sparse sampling and short-range limitations, so as to provide a low-cost and highly adaptable spatial perception solution for mobile vehicles. Summary of the Invention

[0005] The purpose of this invention is to provide a depth estimator based on a monocular RGB camera, which mainly solves the problem of three-dimensional spatial sparsity caused by point-to-point ranging sampling of lidar in the prior art, as well as the problem of nearsightedness of depth cameras, while reducing the cost of the depth estimation system.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A depth estimator based on a monocular RGB camera includes:

[0008] Sensor used to acquire a single RGB image;

[0009] Depth calculation unit, used to estimate depth map from RGB image;

[0010] The output unit is used to parse the generated depth map into signals required by downstream tasks and feed them to downstream modules.

[0011] The depth calculation unit is a spatial probabilistic logic neural network, consisting of an RGB variable assignment function network, an encoder network, and a decoder network. The RGB variable assignment function network is a fully connected neural network composed of several convolutional and pooling layers, used to compress the input RGB image features into a feature vector of length Nr. The encoder network is used to concatenate the input RGB feature variables R, spatial feature variables D, and control variables C to output the posterior probability distribution parameters of the model latent variable Z. The decoder network is used to concatenate the input RGB feature variables R, model latent variables Z, and control variables C to output the probability distribution of the spatial feature variable D.

[0012] Furthermore, in this invention, the depth estimation method of the depth estimator includes establishing a causal effect model and performing inference calculations using the causal effect model; wherein, the specific steps for establishing the causal effect model are as follows:

[0013] S10, Determine the modeling variables for the causal effect model in depth estimation;

[0014] S11, based on the observability of the modeling variables, they are divided into observable variables and hidden variables;

[0015] S12, Perform observation sampling on the observable variables to obtain the observable variable dataset;

[0016] S13, Assign values ​​to the observable variables on the observable variable dataset;

[0017] In step S14, the observable variables that have been assigned values ​​are fed into the spatial probabilistic logistic neural network. The variational inference framework and the maximum lower bound of evidence algorithm are used to iterate the posterior distribution of the hidden variables. At the same time, the parameters of the encoder network, decoder network and RGB variable assignment function network are optimized. After the loss function of the maximum lower bound of evidence algorithm converges, the modeling is completed.

[0018] Further, in step S10, the determined modeling variables include:

[0019] The RGB feature variable R is assigned by observable RGB image features, and its length is Nr;

[0020] The spatial feature variable D, which is directly assigned by encoding observable 3D spatial information, has a length of Nd.

[0021] A control variable C is assigned values ​​by a set of manually assigned control variables [C0, C1, ..., Cn], and its length is Nc.

[0022] The unobservable model latent variable Z has a length of Nz.

[0023] Furthermore, in this invention, the process of inference calculation using a causal effect model is as follows:

[0024] S20, Randomly assign values ​​to the latent variable Z in the model;

[0025] S21, RGB images are acquired through sensors, and RGB feature variables R are assigned values ​​through the trained RGB variable assignment network;

[0026] S22, set the value of the control variable, and directly assign a value to the control variable C;

[0027] S23, input the values ​​of RGB feature variable R, control variable C and model latent variable Z into the decoder network, and output the probability distribution parameters of spatial feature variable D;

[0028] S24, sample from the probability distribution of the spatial feature variable D to obtain the value of the spatial feature variable D, i.e., the space.

[0029] Furthermore, in this invention, the control variable C is a set of control variables set by the user for intervening in the encoder output.

[0030] Furthermore, in this invention, the sensor is a monocular RGB vision camera, and in each depth calculation, the sensor outputs only one monocular RGB image.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] (1) This invention acquires images using a monocular RGB camera and uses a spatial probabilistic logical neural network to model the causal effects between variables, generating a continuous and dense depth map. This effectively overcomes the technical defects of traditional LiDAR, such as "insufficient accuracy of 3D reconstruction due to sparse sampling" and depth cameras, such as "parallax error limitation". The maximum depth estimation distance can reach 1000 meters, supporting spatial perception in ultra-long-distance scenes. It is suitable for scenarios such as UAVs and autonomous driving that require a large-scale environmental understanding.

[0033] (2) This invention uses a variational inference framework and a maximum evidence lower bound algorithm to optimize the model. By compressing the feature vector dimension and simplifying the network structure, the model has a minimum of only 10.3M parameters and a deployment memory of less than 30MB. Compared with the dependence of existing deep learning models on high-performance GPUs, this solution can be directly deployed on low-cost embedded devices such as Raspberry Pi, solving the problems of "high hardware cost and poor adaptability" of traditional technologies.

[0034] (3) By optimizing the encoder-decoder network structure, this invention can quickly respond to dynamic scene changes. Compared with the existing model, which has long inference time and is difficult to adapt to the real-time requirements of mobile vehicles, this solution can provide low-latency spatial perception data for multi-rotor UAVs, wheeled vehicles, etc., ensuring the efficient execution of navigation and obstacle avoidance tasks. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the spatial probabilistic logic network in this invention;

[0036] Figure 2 This is the causal effect model in this invention;

[0037] Figure 3 This is a schematic diagram of the encoder network structure in this invention;

[0038] Figure 4 This is a schematic diagram of the decoder network structure in this invention. Detailed Implementation

[0039] The present invention will be further described below with reference to the accompanying drawings and embodiments. The embodiments of the present invention include, but are not limited to, the following embodiments.

[0040] Example

[0041] The present invention discloses a depth estimator based on a monocular RGB camera, the core of which lies in achieving efficient reconstruction of three-dimensional spatial information through a spatial probabilistic logic neural network of a variational autoencoder.

[0042] In this embodiment, the depth estimator comprises three parts: a sensor, a depth calculation unit, and an output unit. The sensor is a monocular RGB camera used to acquire a single RGB image and transmit it to the depth calculation unit. The depth calculation unit is the core module of the entire system; it processes the input RGB image based on a spatial probabilistic logistic neural network and generates a depth map. The output unit is responsible for parsing the generated depth map into signals required for downstream tasks, such as providing spatial perception data for navigation in multi-rotor drones or wheeled vehicles.

[0043] The core of the depth-of-field calculation unit is a spatial probabilistic logistic neural network, which consists of multiple sub-networks. (See also...) Figure 1The network comprises an RGB variable assignment function network, an encoder network, and a decoder network. The RGB variable assignment function network receives RGB images acquired by a sensor as input and compresses them into a feature vector of length Nr using a fully connected neural network composed of several convolutional and pooling layers. In this embodiment, Nr is set to 192 to balance the accuracy and computational efficiency of feature extraction. In this embodiment, Nc is set to 1, and the control variable C is used to define the cutoff distance for depth estimation, such as 30 meters, 50 meters, 70 meters, and 100 meters.

[0044] The encoder and decoder networks together constitute the core inference mechanism of a spatial probabilistic logic neural network. See also... Figure 3 The encoder network receives RGB feature variables R, control variables C, and spatial feature variables D as input, with a total input vector length of Nr + Nd + Nc. In this embodiment, Nd is equal to the product of the width w and height h of the depth map. In this embodiment, the RGB dimensions are 64*48*3, and the output depth map is 16*12*1. The encoder network calculates the posterior probability distribution parameters of the model's latent variable Z through a multi-layer fully connected network. The length of these parameters is Nz. In this embodiment, Nz is set to 50 to ensure that the model's latent variable Z can fully express the complexity of the causal effects of unknown and difficult-to-observe factors on variable D. See also Figure 4 The decoder network receives RGB feature variables R, control variables C, and model latent variables Z as input, with a total input vector length of Nr + Nc + Nz. The decoder network outputs the probability distribution parameters of the spatial feature variable D of length Nd through a multi-layer fully connected network. In this embodiment, the spatial feature variable D is a depth map with Nd pixels, and the value of each pixel follows a normal distribution. Therefore, the decoder network outputs the mean and variance of each pixel in D.

[0045] Spatial probabilistic logistic neural networks achieve efficient mapping from RGB images to depth maps by modeling causal effects between variables. See also Figure 2 In this implementation, modeling variables are divided into observable variables and latent variables. Among them, the RGB feature variable R, the spatial feature variable D, and the control variable C are observable variables; the model latent variable Z is a latent variable.

[0046] R, C, and Z work together to generate D, but since Z cannot be directly observed, it needs to be indirectly obtained by modeling the posterior probability of Z. In this embodiment, the causal relationship between variables is modeled using the variational inference framework (SVI) and the maximum lower bound of evidence algorithm (ELBO), thereby constructing a complete causal effect model.

[0047] The depth estimation of the depth estimator of this invention includes two processes: a modeling stage and an inference stage. The goal of the modeling stage is to construct a causal effect model among variables. First, based on the observability of the variables, they are divided into observable variables (R, C, D) and latent variables (Z). Then, the observable variables are sampled to generate an observable variable dataset in a simulation environment, containing RGB images, control variable values, and depth maps. In this embodiment, the simulation environment is built using the Unity3D engine, simulating camera poses and lighting conditions under various scenarios. The camera pose is set through the simulation program, and corresponding RGB images and depth maps are generated based on the camera pose and control variables. The above sampling process is repeated several times to generate an observable variable dataset. Subsequently, the RGB variable assignment function network is used to assign values ​​to variable R, and variable D directly takes the depth map encoding value (i.e., the truncated normalized depth value). After the assignment is completed, variables R, C, and D are input into a spatial probabilistic logistic neural network, and the posterior distribution of the latent variable Z is iteratively optimized using a variational inference framework (SVI) and the maximum evidence lower bound algorithm (ELBO). In this embodiment, the ELBO loss function converged after about 2 hours of training, thus completing the construction of the causal effect model.

[0048] The goal of the inference phase is to generate specific spatial information encoding (such as a depth map) given an RGB image and control variable values. First, the latent variable Z is randomly assigned values, and Z variable values ​​are generated by sampling according to a standard normal distribution based on the length Nz of Z. Then, RGB images are acquired through a sensor, and the variable R is assigned values ​​using a pre-trained RGB variable assignment network. Control variable values ​​are set, and variable C is directly assigned a value. In this embodiment, variable C can be defined as the truncation value Ct for depth estimation. For example, when Ct takes values ​​of 30, 50, 70, and 100, it means that the depth estimate is truncated by 30m, 50m, 70m, and 100m, respectively. When Ct is set to these values, the encoder network generates depth maps within a range of 30m, 50m, 70m, and 100m from the current camera. The values ​​of variables R, C, and Z are input into the decoder network, which outputs the probability distribution parameters of the depth map. In this embodiment, the probability distribution of the depth map is assumed to be a normal distribution. Therefore, the decoder network outputs the mean and variance of each pixel to obtain the probability distribution of the depth estimate corresponding to each pixel in the depth map. Finally, sampling is performed on the probability distribution of the depth map or the mean is directly used to obtain the specific depth map.

[0049] The technical advantages of this invention are reflected in several aspects. First, the minimum number of model parameters can reach 10.3 million (10.3M), with GPU usage during training being approximately 200MB to 500MB (FP32 precision), meaning a single consumer-grade graphics card can complete the training. During deployment, only a decoder network needs to be deployed, with the minimum number of parameters as low as 5.14 million (5.14M), and memory requirements of approximately 50MB to 70MB (FP32 precision). In half-precision (FP16) mode, this can be further reduced to below 30MB, supporting deployment on low-cost embedded devices (such as Raspberry Pi). Second, the single-frame inference time is 10 to 50 milliseconds when using only CPU for inference, and as low as 1 to 5 milliseconds when using GPU acceleration. Furthermore, the maximum depth estimation distance can reach 1000 meters, making it suitable for spatial perception modules of mobile vehicles (such as multi-rotor drones, wheeled vehicles, and legged robots). In this embodiment, the depth estimator is applied to the autonomous navigation system of a multi-rotor UAV, which can generate continuous 3D spatial reconstruction data over ultra-long distances in real time, effectively avoiding the "sparse sampling" problem of traditional LiDAR and the "nearsightedness" problem of depth cameras.

[0050] In practical applications, the depth estimator of this invention has wide applicability. For example, in the agricultural field, drones equipped with the depth estimator of this invention can perform 3D modeling of large areas of farmland, providing data support for precision agriculture. In the logistics field, wheeled robots equipped with the depth estimator of this invention can achieve efficient path planning and obstacle avoidance in warehouses. In the security field, legged robots equipped with the depth estimator of this invention can perform patrol tasks in complex terrain, improving security. In this embodiment, the depth estimator is integrated into a wheeled logistics robot, which can quickly generate depth maps in dynamic environments, providing the robot with accurate spatial perception capabilities.

[0051] In summary, this invention overcomes the limitations of traditional methods by introducing variational autoencoders and variational inference algorithms. It has the advantages of low computational requirements, fast inference, and ultra-long-distance depth estimation, and is suitable for spatial perception scenarios of various mobile vehicles.

[0052] The above embodiments are merely one of the preferred embodiments of the present invention and should not be used to limit the scope of protection of the present invention. Any modifications or refinements made to the main design concept and spirit of the present invention that are not of substantial significance, but solve the same technical problem as the present invention, should be included within the scope of protection of the present invention.

Claims

1. A depth estimator based on a monocular RGB camera, characterized in that, include: Sensor used to acquire a single RGB image; Depth calculation unit, used to estimate depth map from RGB image; The output unit is used to parse the generated depth map into signals required by downstream tasks and feed them to downstream modules. The depth calculation unit is a spatial probabilistic logic neural network, consisting of an RGB variable assignment function network, an encoder network, and a decoder network. The RGB variable assignment function network is a fully connected neural network composed of several convolutional and pooling layers, used to compress the input RGB image features into a feature vector of length Nr. The encoder network concatenates the input RGB feature variables R, spatial feature variables D, and control variables C to output the posterior probability distribution parameters of the hidden variable Z. The decoder network concatenates the input RGB feature variables R, control variables C, and hidden variables Z to output the probability distribution parameters of the spatial feature variable D.

2. The depth estimator based on a monocular RGB camera according to claim 1, characterized in that, The depth estimation method of the depth estimator includes establishing a causal effect model and performing inference calculations using the causal effect model; wherein, the specific steps for establishing the causal effect model are as follows: S10, Determine the modeling variables for the causal effect model in depth estimation; S11, based on the observability of the modeling variables, they are divided into observable variables and hidden variables; S12, Perform observation sampling on the observable variables to obtain the observable variable dataset; S13, Assign values ​​to the observable variables on the observable variable dataset; In step S14, the observable variables that have been assigned values ​​are fed into the spatial probabilistic logistic neural network. The variational inference framework and the maximum lower bound of evidence algorithm are used to iterate the posterior distribution of the hidden variables. At the same time, the parameters of the encoder network, decoder network and RGB variable assignment function network are optimized. After the loss function of the maximum lower bound of evidence algorithm converges, the modeling is completed.

3. A depth estimator based on a monocular RGB camera according to claim 2, characterized in that, In step S10, the determined modeling variables include: The RGB feature variable R is assigned by observable RGB image features, and its length is Nr; The spatial feature variable D, which is directly assigned by encoding observable 3D spatial information, has a length of Nd. A control variable C is assigned values ​​by a set of manually assigned control variables [C0, C1, ..., Cn], and its length is Nc. The unobservable model latent variable Z has a length of Nz.

4. A depth estimator based on a monocular RGB camera according to claim 3, characterized in that, The process of inference calculation using the causal effect model is as follows: S20, Randomly assign values ​​to the latent variable Z in the model; S21, RGB images are acquired through sensors, and RGB feature variables R are assigned values ​​through the trained RGB variable assignment network; S22, set the value of the control variable, and directly assign a value to the control variable C; S23, input the values ​​of the above RGB feature variable R, control variable C and model latent variable Z into the decoder network, and output the probability distribution of the spatial feature variable D; S24, sample from the probability distribution of the spatial feature variable D to obtain the value of the spatial feature variable D, i.e., the space.

5. A depth estimator based on a monocular RGB camera according to claim 4, characterized in that, The control variable C is a set of control variables set by the user to intervene in the encoder output.

6. A depth estimator based on a monocular RGB camera according to claim 1, characterized in that, The sensor is a monocular RGB vision camera, and in each depth calculation, the sensor outputs only one monocular RGB image.

Images