Absolute depth estimation method and device and electronic equipment

Abstract

The invention provides an absolute depth estimation method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: performing depth estimation processing on an image shot by a camera through a depth estimation network to obtain relative depths respectively corresponding to a plurality of pixel points in the image; projecting the plurality of pixel points to a three-dimensional space according to the relative depths respectively corresponding to the plurality of pixel points; according to the included angle between the surface normal vector of each pixel point in the three-dimensional space and an ideal ground normal vector, screening the plurality of pixel points to obtain ground points; calculating the estimated height of the camera according to the ground point, and calculating a scale factor according to the estimated height and the actual height of the camera; and according to the scale factor, carrying out scaling processing on the relative depths corresponding to the plurality of pixel points respectively to obtain absolute depths corresponding to the plurality of pixel points respectively. According to the invention, the absolute depth can be accurately estimated without increasing extra cost.

Description

Technical Field

[0001] The present application relates to computer technology, and in particular to an absolute depth estimation method, device, electronic device, computer-readable storage medium and computer program product. Background Art

[0002] Monocular depth estimation refers to the use of images taken by a single camera to estimate the distance (i.e. depth) of each pixel in the image relative to the shooting source (i.e. camera). Monocular depth estimation technology is widely used in robot perception and autonomous driving scenarios. For example, in autonomous driving scenarios, images taken by on-board cameras are used to estimate the depth information of the vehicle's surroundings, thereby realizing autonomous driving functions such as automatic assisted navigation driving and adaptive cruise control.

[0003] Monocular depth estimation can usually only estimate relative depth, not absolute depth, and there is an unknown scale between relative depth and absolute depth. In view of this, in the solutions provided by related technologies, the scale factor is usually calculated using data collected by other sensors. Taking the autonomous driving scenario as an example, the depth obtained by the on-board LiDAR can be used as the absolute depth to calculate the scale factor. However, this solution not only increases the cost, but also complicates the system, bringing about complex joint calibration processes and synchronization issues. Summary of the invention

[0004] The present application provides an absolute depth estimation method, device, electronic device, computer-readable storage medium and computer program product, which can accurately estimate the absolute depth without incurring additional costs.

[0005] The technical solution of this application is implemented as follows:

[0006] The present application provides an absolute depth estimation method, comprising:

[0007] Performing depth estimation processing on the image captured by the camera through a depth estimation network to obtain relative depths corresponding to multiple pixel points in the image;

[0008] Projecting the plurality of pixel points into a three-dimensional space according to relative depths respectively corresponding to the plurality of pixel points;

[0009] According to the angle between the surface normal vector of each pixel point in the three-dimensional space and the ideal ground normal vector, the plurality of pixel points are screened to obtain ground points;

[0010] Calculating an estimated height of the camera according to the ground point, and calculating a scale factor according to the estimated height and the actual height of the camera;

[0011] The relative depths respectively corresponding to the plurality of pixel points are scaled according to the scale factor to obtain the absolute depths respectively corresponding to the plurality of pixel points.

[0012] The present application provides an absolute depth estimation device, comprising:

[0013] A relative depth estimation module is used to perform depth estimation processing on the image taken by the camera through a depth estimation network to obtain relative depths corresponding to multiple pixel points in the image;

[0014] A projection module, configured to project the plurality of pixel points into a three-dimensional space according to relative depths respectively corresponding to the plurality of pixel points;

[0015] A screening module, used to screen the plurality of pixel points to obtain ground points according to the angle between the surface normal vector of each pixel point in the three-dimensional space and the ideal ground normal vector;

[0016] A scale factor calculation module, used to calculate the estimated height of the camera according to the ground point, and calculate the scale factor according to the estimated height and the actual height of the camera;

[0017] The scaling module is used to scale the relative depths corresponding to the multiple pixel points according to the scale factor to obtain the absolute depths corresponding to the multiple pixel points.

[0018] The present application provides an electronic device, including:

[0019] A memory for storing executable instructions;

[0020] The processor is used to implement the absolute depth estimation method provided in the present application when executing the executable instructions stored in the memory.

[0021] The present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute the instructions to implement the absolute depth estimation method provided by the present application.

[0022] The present application provides a computer program product, which includes executable instructions for causing a processor to execute and implement the absolute depth estimation method provided by the present application.

[0023] This application has the following beneficial effects:

[0024] The present application performs depth estimation processing on the image taken by the camera through a depth estimation network to obtain the relative depths corresponding to multiple pixels in the image; the multiple pixels are projected into three-dimensional space according to the relative depths corresponding to the multiple pixels; the multiple pixels are screened and processed to obtain ground points according to the angle between the surface normal vector of each pixel in the three-dimensional space and the ideal ground normal vector; the estimated height of the camera is calculated according to the ground point, and the scale factor is calculated according to the estimated height and actual height of the camera; the relative depths corresponding to the multiple pixels are scaled according to the scale factor to obtain the absolute depths corresponding to the multiple pixels. In summary, the present application, on the basis of estimating the relative depth through the depth estimation network, screens out pixels with surface normal vectors close to the ideal ground normal vector in the three-dimensional space, determines the screened pixels as ground points, and then calculates the estimated height and scale factor according to the ground point. In this way, the geometric constraints are mainly used in the scale recovery process from relative depth to absolute depth, and there is no need to add additional sensors, which can reduce the implementation cost, and at the same time, the absolute depth can be estimated conveniently and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 is a schematic diagram of the architecture of an absolute depth estimation system provided in an embodiment of the present application;

[0027] Figure 2 is a structural diagram of a terminal device provided in an embodiment of the present application;

[0028] Figure 3A is a first flow chart of the absolute depth estimation method provided in an embodiment of the present application;

[0029] Figure 3B is a second flow chart of the absolute depth estimation method provided in an embodiment of the present application;

[0030] Figure 3C is a third flow chart of the absolute depth estimation method provided in an embodiment of the present application;

[0031] Figure 4 is a fourth flow chart of the absolute depth estimation method provided in an embodiment of the present application;

[0032] Figure 5 is a schematic diagram of the architecture of a decoder network provided in an embodiment of the present application;

[0033] Figure 6 is a schematic diagram of a two-dimensional space and a three-dimensional space provided in an embodiment of the present application;

[0034] Figure 7 It is a schematic diagram of ground points in three-dimensional space provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0036] In the following description, reference is made to "some embodiments", which describes a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. In the following description, reference is made to the term "multiple" to mean at least two.

[0037] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0039] The embodiments of the present application provide an absolute depth estimation method, device, electronic device, computer-readable storage medium, and computer program product, which can accurately estimate the absolute depth without adding additional costs. The following describes an exemplary application of the electronic device provided by the embodiments of the present application. The electronic device provided by the embodiments of the present application can be implemented as various types of terminal devices or as a server.

[0040] See also Figure 1 , Figure 1 is a schematic diagram of the architecture of the absolute depth estimation system 100 provided in an embodiment of the present application, wherein the terminal device 400 ( Figure 1 Taking the terminal device 400 deployed in a vehicle as an example, the server 200 is connected via the network 300, wherein the network 300 may be a wide area network or a local area network, or a combination of the two.

[0041] In some embodiments, taking the electronic device as a terminal device as an example, the absolute depth estimation method provided in the embodiment of the present application can be implemented by the terminal device. For example, the terminal device 400 obtains the image taken by the connected camera; performs depth estimation processing on the image through the locally deployed depth estimation network to obtain the relative depths corresponding to multiple pixels in the image; projects multiple pixels into three-dimensional space according to the relative depths corresponding to the multiple pixels; screens multiple pixels according to the angle between the surface normal vector of each pixel in the three-dimensional space and the ideal ground normal vector to obtain ground points; calculates the estimated height of the camera according to the ground points, and calculates the scale factor according to the estimated height and actual height of the camera; scales the relative depths corresponding to the multiple pixels according to the scale factor to obtain the absolute depths corresponding to the multiple pixels. After obtaining the absolute depth, the terminal device 400 can be further applied according to the absolute depth. Taking the autonomous driving scene as an example, the absolute depth can be used for autonomous driving functions such as automatic assisted navigation driving and adaptive cruise, thereby improving the reliability of the autonomous driving function. In the above manner, the terminal device 400 realizes absolute depth estimation locally without transmitting data to a remote server for processing, which can greatly reduce the time and delay of data transmission; since there is no need for data transmission, the risk of privacy data leakage can be reduced; it is not restricted by network conditions, which is especially important for scenarios where the network environment is unstable or cannot connect to the network.

[0042] In some embodiments, taking the electronic device as a server as an example, the absolute depth estimation method provided in the embodiment of the present application can be implemented by the server. For example, the terminal device 400 sends the image captured by the camera to the server 200; the server 200 performs depth estimation processing on the received image through the deployed depth estimation network to obtain the relative depths corresponding to multiple pixel points in the image; the server 200 projects multiple pixel points into a three-dimensional space according to the relative depths corresponding to the multiple pixel points; the server 200 screens the multiple pixel points according to the angle between the surface normal vector of each pixel point in the three-dimensional space and the ideal ground normal vector to obtain the ground point; the server 200 calculates the estimated height of the camera according to the ground point, and calculates the scale factor according to the estimated height and actual height of the camera; the server 200 scales the relative depths corresponding to the multiple pixel points according to the scale factor to obtain the absolute depths corresponding to the multiple pixel points. The server 200 can send the absolute depth to the terminal device 400 so that the terminal device 400 can perform further applications based on the absolute depth. Alternatively, the server 200 can also perform further applications based on the absolute depth. Taking the autonomous driving scenario as an example, the server 200 can obtain the autonomous driving strategy based on the absolute depth planning, and send the autonomous driving strategy to the terminal device 400 so that the terminal device 400 can execute it, thereby realizing the autonomous driving function. In the above method, the server 200 usually has powerful computing power and storage resources, and can efficiently realize absolute depth estimation; the terminal device 400 is mainly responsible for data transmission, and does not need to undertake complex computing tasks, which can reduce the resource consumption of the terminal device 400.

[0043] In some embodiments, the terminal device 400 or the server 200 can implement the absolute depth estimation method provided in the embodiments of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP, and the small program can be controlled by the user to run or close. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0044] In some embodiments, the server 200 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server. The terminal device 400 may be a vehicle-mounted terminal, a mobile phone, a computer, etc., but is not limited thereto. The terminal device 400 and the server 200 may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0045] Taking the electronic device provided in the embodiment of the present application as a terminal device as an example, it can be understood that in the case where the electronic device is a server, Figure 2 Parts of the structure shown in FIG. 1 (such as a user interface, a presentation module, and an input processing module) may be omitted. Figure 2 , Figure 2 is a schematic diagram of the structure of a terminal device 400 provided in an embodiment of the present application, Figure 2 The terminal device 400 shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the terminal device 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not described in detail. Figure 2 Various buses are labeled as bus system 440 .

[0046] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0047] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0048] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0049] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0050] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0051] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0052] A network communication module 452, used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0053] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., display screen, speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripherals and displaying content and information);

[0054] The input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.

[0055] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 An absolute depth estimation device 455 stored in a memory 450 is shown, which may be software in the form of a program or a plug-in, including the following software modules: a relative depth estimation module 4551, a projection module 4552, a screening module 4553, a scale factor calculation module 4554, and a scaling module 4555. These modules are logical, and thus may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0056] The absolute depth estimation method provided in the embodiment of the present application will be described in conjunction with the exemplary application and implementation of the electronic device provided in the embodiment of the present application.

[0057] See also Figure 3A , Figure 3A is a flow chart of an absolute depth estimation method provided in an embodiment of the present application. The absolute depth estimation method provided in an embodiment of the present application can be implemented by an electronic device, such as a terminal device or a server. Figure 3A The steps shown are explained.

[0058] In step 101, a depth estimation network is used to perform depth estimation processing on an image captured by a camera to obtain relative depths corresponding to a plurality of pixel points in the image.

[0059] Here, the image taken by the camera is input into the depth estimation network, and the depth estimation network performs depth estimation on the image. The output of the depth estimation network is a predicted depth map corresponding to the image, which includes the relative depths corresponding to multiple pixel points in the image.

[0060] It is worth noting that the image input to the depth estimation network can be one frame or multiple consecutive frames, which depends on the network architecture of the depth estimation network.

[0061] It is worth noting that the relative depth involved in the embodiment of the present application represents the relative distance between objects represented by different pixels in the image, while the absolute depth represents the actual distance between the object represented by the pixel in the image and the camera.

[0062] In some embodiments, the depth estimation network can be trained in advance, and then the trained depth estimation network is used to perform depth estimation processing on the image captured by the camera to obtain the relative depths corresponding to multiple pixels in the image. In this way, the performance of the depth estimation network can be improved, making the obtained relative depth more accurate.

[0063] In step 102, a plurality of pixel points are projected into a three-dimensional space according to relative depths corresponding to the plurality of pixel points.

[0064] Here, when the relative depth corresponding to each pixel point, the coordinates of each pixel point in two-dimensional space (i.e., the coordinates in the image), and the intrinsic parameter matrix of the camera are known, each pixel point can be projected from the two-dimensional space to the three-dimensional space to obtain the coordinates of each pixel point in the three-dimensional space. It is worth noting that the three-dimensional space involved in the embodiments of the present application takes the camera as the origin of the coordinate system.

[0065] In step 103, a plurality of pixel points are screened to obtain ground points according to the angle between the surface normal vector of each pixel point in the three-dimensional space and the ideal ground normal vector.

[0066] Here, the surface normal vector of each pixel in the three-dimensional space is calculated. Taking any pixel (named as the target pixel) as an example, the surface normal vector of the plane formed by the target pixel and the adjacent pixels can be calculated in the three-dimensional space as the surface normal vector of the target pixel in the three-dimensional space.

[0067] Then, the angle between the surface normal vector of each pixel point in three-dimensional space and the ideal ground normal vector is calculated, where the ideal ground normal vector refers to a vector perpendicular to the ground. Since what needs to be calculated is the angle, the modulus length of the ideal ground normal vector is not limited. For example, the ideal ground normal vector can be a unit vector.

[0068] The smaller the angle between the surface normal vector of a pixel point in three-dimensional space and the ideal ground normal vector, the closer the surface normal vector of the pixel point in three-dimensional space is to perpendicular to the ground. Based on this, several pixel points with the smallest angle can be screened out from multiple pixel points as ground points. For example, pixel points with an angle of zero can be screened out from multiple pixel points as ground points, that is, the surface normal vector of the ground point in three-dimensional space is consistent in direction with the ideal ground normal vector.

[0069] The embodiment of the present application does not limit the method of calculating the angle between two vectors. For example, the two vectors can be substituted into the arccosine function, and the absolute value of the output of the arccosine function can be taken to obtain the numerical representation of the angle between the two vectors.

[0070] It is worth noting that the embodiments of the present application can be well applied to a variety of application scenarios, that is, they can accurately screen out ground points in these application scenarios. Taking the autonomous driving scenario as an example, in the autonomous driving scenario, the camera captures the image of the vehicle's surroundings, the depth difference between different pixels in the image is large, and the actual ground points only account for a part of all pixels. Therefore, by comparing the surface normal vector with the ideal ground normal vector in direction, the ground points can be accurately screened out.

[0071] In some embodiments, the above-mentioned method of screening multiple pixel points to obtain ground points based on the angle between the surface normal vector of each pixel point in three-dimensional space and the ideal ground normal vector can be implemented in the following way: when the angle between the surface normal vector of the target pixel point in three-dimensional space and the ideal ground normal vector is less than the angle threshold, and the vertical coordinate of the target pixel point in three-dimensional space is greater than zero, the target pixel point is determined as a ground point; wherein the target pixel point represents any one of the multiple pixel points.

[0072] Here, since there may be errors in the calculation of the surface normal vector and the camera is installed manually, the vertical axis (y-axis) of its coordinate system may not be strictly perpendicular to the ground. Therefore, when the angle between the surface normal vector of the target pixel point in the three-dimensional space and the ideal ground normal vector is less than the angle threshold, the target pixel point is determined as a ground point, that is, the angle is not strictly constrained to be zero. This can provide a certain degree of fault tolerance and avoid the inability to screen out ground points due to overly strict constraints.

[0073] In the real world, the camera is usually located above the ground. Based on this, the constraint condition can be further limited, that is, when the angle between the surface normal vector of the target pixel point in the three-dimensional space and the ideal ground normal vector is less than the angle threshold, and the vertical coordinate of the target pixel point in the three-dimensional space is greater than zero, the target pixel point is determined as a ground point. In this way, the accuracy of the screening process can be further improved.

[0074] In step 104, an estimated height of the camera is calculated according to the ground point, and a scale factor is calculated according to the estimated height and the actual height of the camera.

[0075] Here, the estimated height of the camera is calculated based on the ground point. For example, when there is only one ground point, the ground point vector of the ground point in three-dimensional space is projected to the direction of the surface normal vector to obtain the estimated height of the camera. The ground point vector refers to a vector in three-dimensional space that starts from the origin of the coordinate system and ends at the ground point.

[0076] Then, the scale factor is calculated according to the estimated height and the actual height of the camera, for example, the estimated height of the camera is divided by the actual height to obtain the scale factor, wherein the actual height of the camera can be known in advance.

[0077] In some embodiments, the number of ground points includes multiple; the above-mentioned calculation of the estimated height of the camera based on the ground points can be achieved in the following way: the ground point vector of each ground point in three-dimensional space is projected to the direction of the surface normal vector to obtain the candidate estimated height corresponding to each ground point; wherein the ground point vector refers to a vector with the origin as the starting point and the ground point as the end point; the estimated height of the camera is determined according to the candidate estimated heights corresponding to multiple ground points.

[0078] Here, when there are multiple ground points, for each ground point, its ground point vector in three-dimensional space is projected to the direction of the surface normal vector to obtain a candidate estimated height. Then, the estimated height of the camera is determined according to the candidate estimated heights corresponding to the multiple ground points. For example, the median of the candidate estimated heights corresponding to the multiple ground points is determined as the estimated height of the camera. Compared with taking the average value, taking the extreme value, etc., taking the median can avoid the influence of abnormal ground points (such as pixels mistaken for ground points) as much as possible, thereby improving the accuracy of the determined estimated height of the camera.

[0079] In step 105, the relative depths corresponding to the plurality of pixel points are scaled according to the scale factor to obtain the absolute depths corresponding to the plurality of pixel points.

[0080] The scale factor can not only reflect the scale difference between the estimated height and the actual height of the camera, but also the scale difference between the relative depth and the absolute depth of the pixel points. Therefore, the relative depths corresponding to the multiple pixels are scaled according to the scale factor. This process is essentially to restore the scale of the relative depths corresponding to the multiple pixels to obtain the absolute depths corresponding to the multiple pixels. For example, the scale factor can be multiplied by the relative depths corresponding to the multiple pixels to obtain the absolute depths corresponding to the multiple pixels.

[0081] After obtaining the absolute depths corresponding to multiple pixel points in the image, further applications can be performed, such as point cloud modeling, or for planning autonomous driving strategies in specific autonomous driving scenarios, without limitation.

[0082] like Figure 3A As shown, the embodiment of the present application performs depth estimation processing on the image taken by the camera through the depth estimation network to obtain the relative depths corresponding to multiple pixels in the image; the multiple pixels are projected into the three-dimensional space according to the relative depths corresponding to the multiple pixels; the multiple pixels are screened and processed to obtain the ground points according to the angle between the surface normal vector of each pixel in the three-dimensional space and the ideal ground normal vector; the estimated height of the camera is calculated according to the ground points, and the scale factor is calculated according to the estimated height and actual height of the camera; the relative depths corresponding to the multiple pixels are scaled according to the scale factor to obtain the absolute depths corresponding to the multiple pixels. In summary, the embodiment of the present application, on the basis of estimating the relative depth through the depth estimation network, screens out pixels with surface normal vectors close to the ideal ground normal vector in the three-dimensional space, determines the screened pixels as ground points, and then calculates the estimated height and scale factor according to the ground points. In this way, the geometric constraints are mainly used in the scale recovery process from relative depth to absolute depth, and no additional sensors are required, which can reduce the implementation cost, and at the same time, the absolute depth can be estimated conveniently and accurately.

[0083] In some embodiments, see Figure 3B , Figure 3B is a flow chart of an absolute depth estimation method provided in an embodiment of the present application, based on Figure 3A Before step 103, steps 201 to 203 may also be performed.

[0084] In step 201, a plurality of adjacent pixel points of a target pixel point are divided into a plurality of adjacent pixel point pairs.

[0085] Here, taking any pixel point in the image (named as the target pixel point) as an example, it is explained how to determine the surface normal vector of the target pixel point in the three-dimensional space.

[0086] For example, two adjacent pixels of a target pixel may be selected, and a surface normal vector of a plane formed by the target pixel and the two adjacent pixels may be calculated in three-dimensional space to serve as the surface normal vector of the target pixel in three-dimensional space.

[0087] However, the accuracy of the surface normal vector determined by the above method depends to a large extent on how the two adjacent pixels are selected, and there is a certain one-sidedness. Therefore, more adjacent pixels can be referenced to determine the surface normal vector of the target pixel in three-dimensional space. First, the multiple adjacent pixels of the target pixel are divided into multiple adjacent pixel pairs, each adjacent pixel pair includes two adjacent pixel points. For example, 8 adjacent pixels in an 8-neighborhood centered on the target pixel can be taken and divided into 4 adjacent pixel pairs, wherein the two adjacent pixels in each adjacent pixel pair can form a right triangle with the target pixel.

[0088] In step 202, a surface normal vector of a plane formed by a target pixel and each adjacent pixel pair is calculated in a three-dimensional space.

[0089] Each pair of adjacent pixel points divided out can form a plane with the target pixel point in three-dimensional space, and the surface normal vector of each constructed plane can be calculated.

[0090] In step 203, vector fusion processing is performed on the surface normal vectors corresponding to the multiple planes to obtain the surface normal vector of the target pixel point in the three-dimensional space.

[0091] Here, the surface normal vectors corresponding to the multiple planes are subjected to vector fusion processing, that is, the surface normal vectors corresponding to the multiple planes are fused into one vector, and the vector is the surface normal vector of the target pixel point in the three-dimensional space. There is no limitation on the method of vector fusion processing, for example, it can be normalization processing + averaging processing.

[0092] like Figure 3B As shown, the embodiment of the present application divides multiple adjacent pixel points of a target pixel point into multiple adjacent pixel point pairs; calculates the surface normal vector of the plane formed by the target pixel point and each adjacent pixel point pair in three-dimensional space; performs vector fusion processing on the surface normal vectors corresponding to the multiple planes respectively, and obtains the surface normal vector of the target pixel point in three-dimensional space. In this way, it is possible to refer to as many adjacent pixel points as possible to improve the accuracy of the surface normal vector of the target pixel point in three-dimensional space.

[0093] In some embodiments, see Figure 3C , Figure 3C is a flow chart of an absolute depth estimation method provided in an embodiment of the present application. Figure 3AStep 101 shown can be implemented through steps 301 to 303 .

[0094] In step 301, the image captured by the camera is encoded through an encoder network to obtain a target feature map corresponding to the image.

[0095] Here, the depth estimation network can adopt an encoder network-decoder network network architecture.

[0096] Based on this, the image captured by the camera is first input into the encoder network, and the encoder network encodes the image to obtain the target feature map of the image. The image input to the encoder network can be one frame or multiple consecutive frames. In the former case, the encoding process implemented by the encoder network can be feature extraction processing; in the latter case, the focus is on the latest frame image in the multiple frames, so the output of the encoder network is the target feature map corresponding to the latest frame image.

[0097] In some embodiments, the camera captures multiple frames of images; the above-mentioned encoding processing of the images captured by the camera through the encoder network can be achieved in the following way: the encoder network performs the following processing: feature extraction processing is performed on the multiple frames of images respectively to obtain feature maps corresponding to the multiple frames of images; the cost volume is calculated according to the feature maps corresponding to the multiple frames of images; the cost volume is spliced ​​with the feature map corresponding to the latest frame image in the multiple frames of images to obtain the target feature map corresponding to the latest frame image.

[0098] Here, the camera captures multiple consecutive frames of images. Based on this, the encoder network can encode the multiple frames to obtain the target feature map corresponding to the latest frame in the multiple frames. The implementation process of the encoding process will be described in detail below.

[0099] First, feature extraction is performed on multiple frames to obtain feature maps corresponding to the multiple frames. Then, cost volume is calculated based on the feature maps corresponding to the multiple frames, where the cost volume is used to measure the similarity between the feature maps corresponding to the multiple frames. Finally, the cost volume is spliced ​​with the feature map corresponding to the latest frame in the multiple frames to obtain the target feature map corresponding to the latest frame. By calculating the cost volume, the feature maps corresponding to the multiple frames can be referenced and effective depth information can be extracted from them, thereby improving the accuracy of the target feature map.

[0100] In step 302, the target feature map is decoded through multiple prediction layers to obtain a predicted depth map output by each prediction layer.

[0101] Here, the decoder network includes multiple cascaded prediction layers, that is, each prediction layer needs to refer to the calculation results of the previous prediction layer during the calculation process, so that features of different scales can be explicitly combined, that is, the semantic information of the deep network is fused into the detail information of the shallow network, so that the decoder network considers object-level reasoning while predicting the depth of the details, and can eliminate the predicted object artifacts. For example, the target feature map is decoded by multiple cascaded prediction layers to obtain the predicted depth map output by each prediction layer, where the predicted depth maps output by different prediction layers correspond to different scales (resolutions).

[0102] In some embodiments, the above-mentioned decoding process of the target feature map through multiple prediction layers to obtain the predicted depth map output by each prediction layer can be implemented in the following manner: the nth prediction layer among the multiple prediction layers performs the following processing: a first convolution process is performed on the nth input feature map to obtain an n+1th input feature map with higher resolution; wherein the first input feature map is the target feature map; a second convolution process is performed on the nth input feature map to obtain an nth intermediate feature map whose number of channels is equal to the target number of channels; feature fusion process is performed on the first intermediate feature map to the nth intermediate feature map to obtain an nth fused feature map; a third convolution process is performed on the nth fused feature map to obtain a predicted depth map output by the nth prediction layer; wherein n is an integer greater than 0.

[0103] Here, the implementation process of the decoding process is explained by taking the nth prediction layer among multiple prediction layers as an example, where n is an integer greater than 0.

[0104] For example, the nth input feature map is subjected to a first convolution process to obtain the n+1th input feature map, wherein the n+1th input feature map has a higher resolution than the nth input feature map, that is, the first convolution process is used to improve the resolution, and it is worth noting that the first input feature map is the target feature map. At the same time, the nth input feature map is subjected to a second convolution process to obtain the nth intermediate feature map whose number of channels is equal to the target number of channels, that is, the second convolution process is used to unify the number of channels for feature fusion processing, wherein the target number of channels can be set according to the actual application scenario, such as being set to 8. Then, the first intermediate feature map to the nth intermediate feature map are subjected to feature fusion processing to obtain the nth fused feature map, for example, the first intermediate feature map to the n-1th intermediate feature map are all subjected to upsampling processing, so that the upsampled first intermediate feature map to the n-1th intermediate feature map have the same resolution as the nth intermediate feature map. Figure 1Then, the nth intermediate feature map is concatenated with the upsampled 1st to n-1th intermediate feature maps to obtain the nth fused feature map. In this way, more accurate object-level inference is introduced into the higher-resolution depth prediction that is less concerned with object-level depth, achieving effective fusion of semantic information and detail information. Finally, the third convolution is performed on the nth fused feature map to obtain the predicted depth map output by the nth prediction layer.

[0105] It is worth noting that the convolution processing involved in the embodiments of the present application (such as the first convolution processing, the second convolution processing and the third convolution processing) is only an example, and there may be other processing before and after the convolution processing, such as activation processing.

[0106] In step 303, the depths respectively corresponding to the multiple pixels in the predicted depth map output by the last prediction layer are determined as the relative depths respectively corresponding to the multiple pixels in the image.

[0107] Through the decoding processing realized by the cascaded multiple prediction layers, the predicted depth map output by the last prediction layer contains accurate and effective depth information. Therefore, the depths corresponding to the multiple pixels in the predicted depth map output by the last prediction layer are determined as the relative depths corresponding to the multiple pixels in the image.

[0108] like Figure 3C As shown, the embodiment of the present application encodes the image captured by the camera through an encoder network to obtain a target feature map corresponding to the image; decodes the target feature map through multiple prediction layers to obtain a predicted depth map output by each prediction layer; and determines the depths corresponding to multiple pixels in the predicted depth map output by the last prediction layer as the relative depths corresponding to multiple pixels in the image. In this way, through a cascaded processing mechanism, the semantic information of the deep network can be integrated into the detailed information of the shallow network, thereby improving the accuracy of the relative depth obtained.

[0109] Below, an exemplary application of the embodiment of the present application in a practical application scenario will be described. As an example, the embodiment of the present application provides the following Figure 4 A flowchart of the absolute depth estimation method shown in FIG. Figure 4 , explained in step form.

[0110] Step 1: Input the two frames of images taken by the camera into the depth estimation network to obtain the relative depths corresponding to multiple pixel points in the current frame image.

[0111] Here, the current frame image (corresponding to the latest frame image above) and the previous frame image taken by the camera (such as a car-mounted monocular camera) are obtained, and the current frame image and the previous frame image are input into the encoder network in the depth estimation network. In the encoder network, the feature map corresponding to the current frame image and the feature map corresponding to the previous frame image are extracted respectively, and the cost volume is calculated based on the two feature maps. Then, the cost volume and the feature map corresponding to the current frame image are spliced ​​to obtain the target feature map corresponding to the current frame image.

[0112] Then, the target feature map corresponding to the current frame image is input into the decoder network in the depth estimation network. In the decoder network, the cascade prediction (CP) mechanism is used to implement the decoding process. The cascade prediction mechanism explicitly combines features of different scales, that is, it fuses the semantic information of the deep network into the detail information of the shallow network, so that the decoder network considers object-level reasoning while predicting the depth of the details, eliminating the predicted object artifacts. The fusion of semantic information can enable the decoder network to learn that some details are part of the object, so that the predicted depth of the details is consistent with the object as a whole. The network architecture of the decoder network is as follows: Figure 5 As shown in , the decoder network includes multiple cascaded prediction layers, each of which uses a convolutional layer to unify the number of channels of the feature map to 8 (corresponding to the second convolution process above), so that feature maps of different resolutions correspond to a unified number of channels, simplifying subsequent calculations, that is, low-resolution feature maps can be upsampled and spliced ​​to high-resolution feature maps. By fusing semantic information and detail information, the decoder network introduces more accurate object-level inference into higher-resolution depth predictions that were initially less concerned with object-level depth. Figure 5 As shown, each prediction layer of the decoder network outputs a predicted depth map (or relative depth map), and the predicted depth map with the largest resolution is taken from these predicted depth maps. rd (i.e., the predicted depth map output by the last prediction layer), and execute step 2, where I rd Included are the relative depths corresponding to multiple pixel points in the current frame image.

[0113] Step 2: Calculate the surface normal vector pixel by pixel based on the relative depths corresponding to multiple pixels in the current frame image.

[0114] First, all pixels in the current frame image are projected into three-dimensional space according to the camera model. The projection formula is as follows:

[0115] D(p i,j )·p i,j =K·P i,j

[0116] Among them, p i,j =[x,y,1] T , represents the coordinates of a pixel point p in the xth row and yth column of the current frame image in two-dimensional space under homogeneous coordinates; P i,j =[X,Y,Z] T Represents pixel p i,j Coordinates in three-dimensional space; D(p i,j ) is the pixel p i,j The relative depth of the camera; K represents the intrinsic parameter matrix of the camera.

[0117] like Figure 6 As shown in the figure, for each pixel in the current frame image, the 8-neighborhood convention method is used to determine the planes where it is located in the three-dimensional space. For example, for a certain pixel p i,j , the surrounding 8 adjacent pixels are divided into 4 pairs (in Figure 6 Each pair is represented by the same pattern), pixel p i,j can be connected to form a pair of two adjacent pixels to form a right triangle, that is, the divided adjacent pixel pairs can be expressed as L(p i,j )={[p i+1,j ,p i,j+1 ],[p i+1,j+1 ,p i-1,j+1 ]..........}. For each adjacent pixel pair, it can be connected to the pixel point p in three-dimensional space. i,j Together they form a plane, and then the surface normal vector m of the plane can be calculated l , the calculation formula is as follows:

[0118]

[0119] Among them, L a,b Represents the bth pixel in the ath adjacent pixel pair, l = 1, 2, 3, 4.

[0120] Pixel p i,j The surface normal vector M(P i,j ) is obtained by normalizing and averaging the surface normal vectors corresponding to the four planes. The calculation formula is as follows:

[0121]

[0122] Step 3: Detect ground points based on the surface normal vector corresponding to each pixel.

[0123] The ground points are selected based on the surface normal vector generated in step 2. A ground point refers to a pixel point with a surface normal vector close to the ideal ground normal vector in direction. Based on the ideal ground normal vector and the calculated surface normal vector, the embodiment of the present application uses a function s(P) based on the inverse cosine function. i,j ) to detect ground points, function s(P i,j ) is calculated as follows:

[0124]

[0125] Where · represents the inner product operation; N represents the ideal ground normal vector, such as [0,1,0] T ; S corresponds to the angle mentioned above. It is worth noting that since the calculation of the surface normal vector corresponding to the pixel point may have errors and certain uncertainties, and the camera is installed manually, the y-axis of the camera coordinate system (i.e., three-dimensional space) may not be strictly perpendicular to the ground, such as Figure 7 As shown, Figure 7 The origin (point O) shown is the coordinate of the camera. Therefore, the embodiment of the present application sets a threshold T max , for S <T max , and y(P i,j )>0 is considered as a ground point, that is, ground point A={P i,j |s(P i,j ) <T max ,y(P i,j )>0}, where y(P i,j ) represents pixel p i,j The value on the y-axis of the camera coordinate system.

[0126] Step 4: Calculate the estimated height of the camera based on the ground point.

[0127] When multiple ground points are identified from the current frame image, the geometric relationship between the ground points and the camera can be used. Figure 7 As shown, for the ground point p i，j For example, the ground point vector in three-dimensional space is Projected onto the surface normal vector M(P i,j ) direction, and get the ground point p i,j The corresponding candidate estimated height h(P i,j ), the calculation formula is as follows:

[0128]

[0129] in,

[0130] When the current frame image includes multiple ground points, candidate estimated heights corresponding to the multiple ground points are obtained. In the embodiment of the present application, the median h of all candidate estimated heights is selected. p As the estimated height of the camera, taking the median can eliminate the adverse effects of abnormal ground points as much as possible.

[0131] Step 5: Calculate the scale factor based on the estimated height of the camera and the actual height, and use the scale factor to restore the relative depth to the absolute depth.

[0132] Here, get the actual height h of the camera gt And calculate the scale factor sf, the calculation formula is as follows:

[0133]

[0134] After successfully obtaining the scale factor sf, the depth map I is predicted rd Convert to absolute depth map I ad , the calculation formula is as follows:

[0135] I ad =sfI rd

[0136] Among them, the absolute depth map I ad Includes the absolute depths corresponding to multiple pixels in the current frame image.

[0137] After obtaining the absolute depth map I ad Afterwards, the absolute depth map I ad For further application, for example, in the scenario of autonomous driving, the absolute depth map I ad To realize the autonomous driving function.

[0138] The embodiments of the present application have at least the following technical effects:

[0139] 1) A new cascade prediction (CP) mechanism is used to provide better object-level reasoning, so that object boundaries can be better distinguished in depth maps and the depth reasoning of objects is more complete and consistent.

[0140] 2) For autonomous driving scenarios, the estimated height of the camera is calculated during inference to implement geometric constraints so that the absolute depth can be accurately restored without adding additional cost to the autonomous vehicle.

[0141] The following is a description of an exemplary structure of the absolute depth estimation device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, for example, Figure 2As shown, the software modules stored in the absolute depth estimation device 455 of the memory 450 may include: a relative depth estimation module 4551, which is used to perform depth estimation processing on the image taken by the camera through a depth estimation network to obtain relative depths corresponding to multiple pixel points in the image; a projection module 4552, which is used to project multiple pixel points into a three-dimensional space according to the relative depths corresponding to the multiple pixel points; a screening module 4553, which is used to screen multiple pixel points according to the angle between the surface normal vector of each pixel point in the three-dimensional space and the ideal ground normal vector to obtain ground points; a scale factor calculation module 4554, which is used to calculate the estimated height of the camera according to the ground points, and calculate the scale factor according to the estimated height and the actual height of the camera; a scaling module 4555, which is used to scale the relative depths corresponding to the multiple pixel points according to the scale factor to obtain the absolute depths corresponding to the multiple pixel points.

[0142] In some embodiments, the absolute depth estimation device 455 also includes a surface normal vector calculation module, which is used to: divide multiple adjacent pixel points of the target pixel point into multiple adjacent pixel point pairs; calculate the surface normal vector of the plane formed by the target pixel point and each adjacent pixel point pair in three-dimensional space; perform vector fusion processing on the surface normal vectors corresponding to the multiple planes to obtain the surface normal vector of the target pixel point in the three-dimensional space; wherein the target pixel point represents any one of the multiple pixel points.

[0143] In some embodiments, the screening module 4553 is also used to: when the angle between the surface normal vector of the target pixel point in the three-dimensional space and the ideal ground normal vector is less than the angle threshold, and the vertical coordinate of the target pixel point in the three-dimensional space is greater than zero, determine the target pixel point as a ground point; wherein the target pixel point represents any one of multiple pixel points.

[0144] In some embodiments, the number of ground points includes multiple; the scale factor calculation module 4554 is also used to: project the ground point vector of each ground point in the three-dimensional space to the direction of the surface normal vector to obtain the candidate estimated height corresponding to each ground point; wherein the ground point vector refers to a vector with the origin as the starting point and the ground point as the end point; determine the estimated height of the camera according to the candidate estimated heights corresponding to multiple ground points.

[0145] In some embodiments, the depth estimation network includes an encoder network and a decoder network, and the decoder network includes multiple cascaded prediction layers; the relative depth estimation module 4551 is also used to: encode the image taken by the camera through the encoder network to obtain a target feature map corresponding to the image; decode the target feature map through multiple prediction layers to obtain a predicted depth map output by each prediction layer; and determine the depths corresponding to multiple pixels in the predicted depth map output by the last prediction layer as the relative depths corresponding to multiple pixels in the image.

[0146] In some embodiments, the relative depth estimation module 4551 is also used to perform the following processing through the nth prediction layer among multiple prediction layers: performing a first convolution processing on the nth input feature map to obtain an n+1th input feature map with higher resolution; wherein the 1st input feature map is the target feature map; performing a second convolution processing on the nth input feature map to obtain an nth intermediate feature map whose number of channels is equal to the target number of channels; performing feature fusion processing on the 1st intermediate feature map to the nth intermediate feature map to obtain an nth fused feature map; performing a third convolution processing on the nth fused feature map to obtain a predicted depth map output by the nth prediction layer; wherein n is an integer greater than 0.

[0147] In some embodiments, the camera captures multiple frames of images; the relative depth estimation module 4551 is also used to perform the following processing through the encoder network: perform feature extraction processing on the multiple frames of images respectively to obtain feature maps corresponding to the multiple frames of images; calculate the cost volume according to the feature maps corresponding to the multiple frames of images; splice the cost volume with the feature map corresponding to the latest frame image in the multiple frames of images to obtain the target feature map corresponding to the latest frame image.

[0148] The embodiment of the present application provides a computer program product or a computer program, which includes executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the executable instructions from the computer-readable storage medium, and the processor executes the executable instructions, so that the electronic device performs the absolute depth estimation method described in the embodiment of the present application.

[0149] An embodiment of the present application provides a computer-readable storage medium storing executable instructions, wherein executable instructions are stored. When the executable instructions are executed by a processor, the processor will be caused to execute the absolute depth estimation method provided by the embodiment of the present application.

[0150] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0151] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0152] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0153] As an example, the executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0154] The above are only embodiments of the present application and are not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. An absolute depth estimation method, characterized in that: include: Performing depth estimation processing on the image captured by the camera through a depth estimation network to obtain relative depths corresponding to multiple pixel points in the image; Projecting the plurality of pixel points into a three-dimensional space according to relative depths respectively corresponding to the plurality of pixel points; According to the angle between the surface normal vector of each pixel point in the three-dimensional space and the ideal ground normal vector, the plurality of pixel points are screened to obtain ground points; Calculating an estimated height of the camera according to the ground point, and calculating a scale factor according to the estimated height and the actual height of the camera; The relative depths respectively corresponding to the plurality of pixel points are scaled according to the scale factor to obtain the absolute depths respectively corresponding to the plurality of pixel points.

2. The method according to claim 1, characterized in that Before calculating the angle between the surface normal vector of each pixel point in the three-dimensional space and the ideal ground normal vector, the method further comprises: Divide a plurality of adjacent pixel points of a target pixel point into a plurality of adjacent pixel point pairs; Calculating a surface normal vector of a plane formed by the target pixel and each adjacent pixel pair in three-dimensional space; Performing vector fusion processing on surface normal vectors corresponding to multiple planes respectively to obtain the surface normal vector of the target pixel point in three-dimensional space; The target pixel point represents any one of the multiple pixel points.

3. The method according to claim 1, characterized in that The method of screening the plurality of pixel points to obtain ground points according to the angle between the surface normal vector of each pixel point in the three-dimensional space and the ideal ground normal vector comprises: When the angle between the surface normal vector of the target pixel point in the three-dimensional space and the ideal ground normal vector is less than the angle threshold, and the ordinate of the target pixel point in the three-dimensional space is greater than zero, the target pixel point is determined as a ground point; The target pixel point represents any one of the multiple pixel points.

4. The method according to claim 1, characterized in that: The number of the ground points includes a plurality; and the calculating the estimated height of the camera according to the ground points includes: Project the ground point vector of each ground point in three-dimensional space to the direction of the surface normal vector to obtain the candidate estimated height corresponding to each ground point; wherein the ground point vector refers to a vector formed by taking the origin as the starting point and the ground point as the end point; The estimated height of the camera is determined according to candidate estimated heights respectively corresponding to a plurality of ground points.

5. The method according to any one of claims 1 to 4, characterized in that: The depth estimation network includes an encoder network and a decoder network, and the decoder network includes a plurality of cascaded prediction layers; the depth estimation network performs depth estimation processing on an image captured by a camera to obtain relative depths corresponding to a plurality of pixel points in the image, including: The image captured by the camera is encoded by the encoder network to obtain a target feature map corresponding to the image; Decoding the target feature map through the multiple prediction layers to obtain a predicted depth map output by each prediction layer; The depths respectively corresponding to the multiple pixels in the predicted depth map output by the last prediction layer are determined as the relative depths respectively corresponding to the multiple pixels in the image.

6. The method according to claim 5, characterized in that The decoding process of the target feature map through the multiple prediction layers to obtain a predicted depth map output by each prediction layer includes: The following process is performed by the nth prediction layer in the plurality of prediction layers: Performing a first convolution process on the nth input feature map to obtain an n+1th input feature map with a higher resolution; wherein the first input feature map is the target feature map; Perform a second convolution on the nth input feature map to obtain an nth intermediate feature map whose number of channels is equal to the number of target channels; Perform feature fusion processing on the first intermediate feature map to the nth intermediate feature map to obtain the nth fused feature map; Perform the third convolution process on the nth fused feature map to obtain the predicted depth map output by the nth prediction layer; Wherein, n is an integer greater than 0.

7. The method according to claim 5, characterized in that The camera captures multiple frames of images; the encoding process of the images captured by the camera through the encoder network to obtain a target feature map corresponding to the image includes: The following processing is performed by the encoder network: Performing feature extraction processing on the multiple frames of images respectively to obtain feature maps corresponding to the multiple frames of images respectively; Calculate the cost volume according to the feature maps corresponding to the multiple frames of images respectively; The cost volume is spliced ​​with a feature map corresponding to a latest frame image in the multiple frames to obtain a target feature map corresponding to the latest frame image.

8. An absolute depth estimation device, characterized in that: include: A relative depth estimation module is used to perform depth estimation processing on the image taken by the camera through a depth estimation network to obtain relative depths corresponding to multiple pixel points in the image; A projection module, configured to project the plurality of pixel points into a three-dimensional space according to relative depths respectively corresponding to the plurality of pixel points; A screening module, used to screen the plurality of pixel points to obtain ground points according to the angle between the surface normal vector of each pixel point in the three-dimensional space and the ideal ground normal vector; A scale factor calculation module, used to calculate the estimated height of the camera according to the ground point, and calculate the scale factor according to the estimated height and the actual height of the camera; The scaling module is used to scale the relative depths corresponding to the multiple pixel points according to the scale factor to obtain the absolute depths corresponding to the multiple pixel points.

9. An electronic device, characterized in that: include: A memory for storing executable instructions; A processor, configured to implement the method according to any one of claims 1 to 7 when executing the executable instructions stored in the memory.

10. A computer-readable storage medium, characterized in that: Executable instructions are stored, and when executed by a processor, the method described in any one of claims 1 to 7 is implemented.

Images