Monocular image depth estimation method and apparatus, and computer device

Abstract

A monocular image depth estimation method and apparatus (400), and a computer device (500). The method comprises: obtaining first depth maps of images to be estimated; obtaining a dynamic point set and a first pose transformation result of millimeter wave point clouds to be estimated; obtaining a second depth map of the next frame of image to be estimated; calculating a projection error between the first depth map and the second depth map of the next frame of image to be estimated; obtaining a second pose transformation result of the millimeter wave point clouds to be estimated; obtaining a pose estimation error; calculating a depth error between moving objects in the two frames of images to be estimated; and obtaining overall training loss, obtaining a complete depth estimation model, and performing monocular image depth estimation on the images to be estimated.

Description

A method, device and computer equipment for estimating depth of monocular image

[0001] Related applications

[0002] This application claims priority to Chinese patent application number 202311050584.9, filed on August 21, 2023, entitled “A method, device and computer equipment for monocular image depth estimation”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of image processing technology, and in particular to a method, apparatus, and computer device for estimating depth from a monocular image. Background Art

[0004] Three-dimensional environmental perception is a critical technology in the fields of mobile robots and unmanned vehicles. Currently, this technology relies primarily on expensive, dense LiDAR sensors to obtain accurate 3D information about the environment. Compared to dense LiDAR sensors, self-supervised monocular image depth estimation uses camera-based images to perceive depth without relying on additional depth markers, offering significant cost advantages.

[0005] However, estimating depth directly from an image is an ill-posed problem, and mainstream deep learning methods cannot accurately estimate the absolute depth of an image using only images captured by the camera. Therefore, related technologies enhance the ability to estimate image depth by introducing additional inexpensive modal data. Specifically, depth estimation can be assisted by introducing GPS (Global Positioning System) coordinates, IMU (Inertial Measurement Unit) data, or sparse LiDAR data. However, assisting depth estimation by introducing GPS coordinates, IMU data, or sparse LiDAR data requires removing moving objects from the image and relies on the static environment assumption, resulting in a large jitter in the depth estimation results of two adjacent monocular images with moving objects, and the stability of the image depth estimation results cannot be guaranteed.

[0006] The problem that the relevant technology cannot guarantee the stability of the depth estimation results of the image has not yet been solved.

[0007] Summary of the Invention

[0008] According to various embodiments of the present application, a method, apparatus, and computer device for estimating depth from a monocular image are provided.

[0009] In a first aspect, the present application provides a method for estimating depth from a monocular image. The method comprises the following steps:

[0010] Using a preset initial depth estimation model, perform depth estimation on two frames of the image to be estimated to obtain a first depth map of the image to be estimated; the first depth map of the image to be estimated includes the first depth map of the previous frame of the image to be estimated and the first depth map of the next frame of the image to be estimated;

[0011] Using a preset initial point cloud estimation model, point cloud estimation is performed on two frames of millimeter-wave point clouds to be estimated corresponding to the two frames of the images to be estimated, to obtain a dynamic point set and a first pose transformation result of the millimeter-wave point clouds to be estimated;

[0012] Based on the first pose transformation result, calculating the extrinsic parameter transformation value of the camera; based on the extrinsic parameter transformation value of the camera and the intrinsic parameter value of the camera, projecting the first depth map of the previous frame of the picture to be estimated to the perspective of the next frame of the picture to be estimated to obtain a second depth map of the next frame of the picture to be estimated; calculating the projection error between the first depth map of the next frame of the picture to be estimated and the second depth map of the next frame of the picture to be estimated according to a preset projection error calculation method;

[0013] Using a preset estimation algorithm, perform overall pose transformation estimation on the two frames of the millimeter-wave point cloud to be estimated, and obtain a second pose transformation result of the overall pose transformation of the millimeter-wave point cloud to be estimated; based on the first pose transformation result and the second pose transformation result, obtain a pose estimation error between the first pose transformation result and the second pose transformation result according to a preset pose estimation error calculation method;

[0014] Calculating the depth error of the moving object in the two frames of the to-be-estimated pictures based on the first depth map and the dynamic point set according to a preset method for calculating the depth error of the moving object;

[0015] Obtaining an overall training loss for the picture to be estimated based on a projection error between a first depth map of the subsequent frame of the picture to be estimated and a second depth map of the subsequent frame of the picture to be estimated, a pose estimation error between the first pose transformation result and the second pose transformation result, and a depth error of the moving object in the two frames of the pictures to be estimated, and training the initial depth estimation model and the initial point cloud estimation model using the overall training loss until the initial depth estimation model and the initial point cloud estimation model converge, thereby obtaining a complete depth estimation model for monocular image depth estimation;

[0016] Based on the complete depth estimation model, monocular image depth estimation is performed on the image to be estimated.

[0017] In one embodiment, before performing depth estimation on two frames of images to be estimated using a preset initial depth estimation model to obtain a first depth map of the images to be estimated, the following steps are included:

[0018] Perform mean subtraction and variance division operations on the two original frames to be estimated to generate two first frames;

[0019] The two frames of the first pictures are scaled to a preset size using a preset scaling method to obtain the scaled two frames of the to-be-estimated pictures.

[0020] In one embodiment, the method of performing depth estimation on two frames of images to be estimated using a preset initial depth estimation model to obtain a first depth map of the images to be estimated includes the following steps:

[0021] Obtaining depth features of the image to be estimated using a preset depth coding network of the initial depth estimation model;

[0022] Using the preset depth decoding network of the initial depth estimation model, extracting features related to depth estimation from the acquired depth features to obtain an inverse depth map of the image to be estimated;

[0023] The inverse depth map is processed by taking the inverse thereof to obtain a first depth map of the image to be estimated.

[0024] In one embodiment, the method of using a preset initial point cloud estimation model to perform point cloud estimation on two frames of millimeter wave point clouds to be estimated corresponding to the two frames of the images to be estimated, and obtaining a dynamic point set and a first pose transformation result of the millimeter wave point clouds to be estimated, includes the following steps:

[0025] Obtaining the scene flow of the millimeter-wave point cloud to be estimated by using a preset scene flow prediction network of the initial point cloud estimation model;

[0026] According to a preset dynamic point screening condition, dynamic points in the scene stream that satisfy a translation deviation greater than or equal to one times the variance of the average translation offset are screened out to obtain a dynamic point set of the millimeter wave point cloud to be estimated;

[0027] Using the preset pose estimation network of the initial point cloud estimation model, obtaining a matrix of one row and six columns of the millimeter wave point cloud to be estimated;

[0028] The matrix with one row and six columns is converted into a matrix with three rows and four columns by using a preset rotation formula, so as to obtain the first posture transformation result of the millimeter wave point cloud to be estimated.

[0029] In one embodiment, the step of calculating the camera's extrinsic parameter transformation value based on the first pose transformation result; and projecting the first depth map of the previous frame of the image to be estimated to the perspective of the next frame of the image to be estimated based on the camera's extrinsic parameter transformation value and the camera's intrinsic parameter value to obtain the second depth map of the next frame of the image to be estimated comprises the following steps:

[0030] Based on the first pose transformation result and the preset extrinsic parameter values ​​of the millimeter wave radar to the camera, obtaining the extrinsic parameter transformation values ​​of the camera corresponding to the two frames of the images to be estimated;

[0031] Based on the extrinsic parameter transformation value of the camera and the intrinsic parameter value of the camera, the first depth map of the previous frame of the picture to be estimated is projected to the viewing angle of the next frame of the picture to be estimated, and based on the projection result, the second depth map of the next frame of the picture to be estimated is obtained.

[0032] In one embodiment, the calculation formula of the projection error L1 between the first depth map of the next frame to be estimated picture and the second depth map of the next frame to be estimated picture is:

[0033] Among them, D T Denotes the first depth map of the image to be estimated at time T, D T-1→T It represents the second depth map of the image to be estimated at time T obtained by projecting the first depth map of the image to be estimated at time T-1 to the viewing angle of the image to be estimated at time T. SSIM represents the loss of projection error. are pre-set parameters.

[0034] In one embodiment, the method of using a preset estimation algorithm to perform overall pose transformation estimation on two frames of the millimeter wave point cloud to be estimated to obtain a second pose transformation result of the overall pose transformation of the millimeter wave point cloud includes:

[0035] The ICP algorithm is used to estimate the overall pose transformation of the two frames of the millimeter-wave point cloud to be estimated, and a second pose transformation result of the overall pose transformation of the millimeter-wave point cloud to be estimated is obtained.

[0036] In one embodiment, the calculation formula of the pose estimation error L2 between the first pose transformation result and the second pose transformation result is:

[0037] Among them, TR1 represents the first posture transformation result, and TR2 represents the second posture transformation result.

[0038] In one embodiment, the calculation formula of the depth error L3 of the moving object in the two frames of the to-be-estimated pictures is:

[0039] Among them, RD T-1 Represents the dynamic point set at time T-1, RD T represents the dynamic point set at time T, p and q represent RD T-1 and RD T For any pair of points in , Sim(p,q) represents the probability that p and q come from the same obstacle, π(cos(Loc q , TR T-1→T [:,3])) is the indicator function, Loc q represents the three-dimensional space coordinates of q, TR T-1→T represents the first pose transformation result of the millimeter wave point cloud to be estimated from time T-1 to time T, π(D T (p)<D T-1 (q)) is the indicator function, D T (p) represents the depth value of p in the first depth map of the image to be estimated at time T, D T-1 (q) represents the depth value of q in the first depth map of the image to be estimated at time T-1.

[0040] In one embodiment, the overall training loss formula L of the image to be estimated is: L = L1 + β·L2 + (1-β)·L3

[0041] in, ep represents the current training epoch, and Max_epoch represents the maximum training epoch.

[0042] In a second aspect, the present application further provides a monocular image depth estimation device, comprising: a depth estimation module, a point cloud estimation module, a first calculation module, a second calculation module, a third calculation module, a training module, and an estimation module.

[0043] The depth estimation module is used to use a preset initial depth estimation model to perform depth estimation on two frames of images to be estimated, and obtain a first depth map of the images to be estimated; the first depth map of the images to be estimated includes the first depth map of the previous frame of the image to be estimated and the first depth map of the next frame of the image to be estimated.

[0044] The point cloud estimation module is used to use a preset initial point cloud estimation model to perform point cloud estimation on two frames of millimeter wave point clouds to be estimated corresponding to the two frames of the pictures to be estimated, and obtain the dynamic point set and the first pose transformation result of the millimeter wave point clouds to be estimated.

[0045] A first calculation module is used to calculate the camera's extrinsic parameter transformation value based on the first pose transformation result; based on the camera's extrinsic parameter transformation value and the camera's intrinsic parameter value, project the first depth map of the previous frame of the picture to be estimated to the viewing angle of the next frame of the picture to be estimated to obtain the second depth map of the next frame of the picture to be estimated; and calculate the projection error between the first depth map of the next frame of the picture to be estimated and the second depth map of the next frame of the picture to be estimated according to a preset projection error calculation method.

[0046] The second calculation module is used to use a preset estimation algorithm to perform overall pose transformation estimation on the two frames of the millimeter-wave point cloud to be estimated, and obtain a second pose transformation result of the overall pose transformation of the millimeter-wave point cloud to be estimated; based on the first pose transformation result and the second pose transformation result, according to a preset pose estimation error calculation method, obtain the pose estimation error of the first pose transformation result and the second pose transformation result.

[0047] The third calculation module is used to calculate the depth error of the moving object in the two frames of the to-be-estimated pictures based on the first depth map and the dynamic point set and according to a preset method for calculating the depth error of the moving object.

[0048] A training module is used to obtain an overall training loss of the picture to be estimated based on the projection error of the first depth map of the next frame of the picture to be estimated and the second depth map of the next frame of the picture to be estimated, the pose estimation error of the first pose transformation result and the second pose transformation result, and the depth error of the moving object in the two frames of the pictures to be estimated, and use the overall training loss to train the initial depth estimation model and the initial point cloud estimation model until the initial depth estimation model and the initial point cloud estimation model converge, so as to obtain a complete depth estimation model for monocular image depth estimation.

[0049] And an estimation module, for performing monocular image depth estimation on the image to be estimated based on the complete depth estimation model.

[0050] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the monocular image depth estimation method described in the first aspect when executing the computer program.

[0051] The details of one or more embodiments of the present application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the present application will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the disclosed drawings without any creative work.

[0053] FIG1 is a block diagram of the hardware structure of a terminal of a monocular image depth estimation method according to some embodiments.

[0054] FIG2 is a flowchart of a method for estimating depth from a monocular image according to some embodiments.

[0055] FIG3 is a flowchart of a method for estimating depth from a monocular image according to some embodiments.

[0056] FIG4 is a block diagram of a monocular image depth estimation device according to some embodiments.

[0057] FIG5 is a block diagram of a computer device according to some embodiments. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0059] Unless otherwise defined, the technical terms or scientific terms involved in this application should have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "an", "a", "the", "these" and the like in this application do not indicate quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "plurality" involved in this application refers to two or more. "And / or" describes the relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, and B exists alone. Generally, the character " / " indicates that the related objects are in an "or" relationship. The terms "first," "second," "third," etc. used in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0060] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, when running on a terminal, FIG1 is a hardware structure block diagram of a terminal 100 of the monocular image depth estimation method of this embodiment. As shown in FIG1 , the terminal 100 may include one or more (only one is shown in FIG1 ) processors 102 and a memory 104 for storing data, wherein the processor 102 may include but is not limited to a processing device such as a microprocessor MCU (Microcontroller Unit) or a programmable logic device FPGA (Field Programmable Gate Array). The terminal 100 may also include a transmission device 106 and an input and output device 108 for communication functions. It will be understood by those skilled in the art that the structure shown in FIG1 is only for illustration and does not limit the structure of the terminal 100. For example, the terminal 100 may also include more or fewer components than those shown in FIG1 , or have a different configuration than that shown in FIG1 .

[0061] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the monocular image depth estimation method provided in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a remote memory remotely located relative to the processor 102, and these remote memories may be connected to the terminal 100 via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0062] Transmission device 106 is used to receive or transmit data via a network. The aforementioned network includes a wireless network provided by the telecommunications provider of terminal 100. In one embodiment, transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0063] In this embodiment, a monocular image depth estimation method is provided. FIG2 is a flow chart of the monocular image depth estimation method of this embodiment. As shown in FIG2 , the process includes the following steps S210 to S270 .

[0064] Step S210 , using a preset initial depth estimation model, performs depth estimation on two frames of images to be estimated to obtain a first depth map of the images to be estimated; the first depth map of the images to be estimated includes the first depth map of the previous frame of the image to be estimated and the first depth map of the next frame of the image to be estimated.

[0065] In this step, the above-mentioned initial depth estimation model may include a depth encoding network and a depth decoding network. Among them, the depth encoding network can be used to extract depth features of the image to be estimated. The depth decoding network can be used to extract features related to depth estimation of the acquired depth features. Specifically, the depth decoding network can pre-set multiple convolution layers, and each convolution layer is followed by an upsampling layer, and multiple convolution layers and upsampling layers are used to realize feature extraction related to depth estimation of the acquired depth features. The above-mentioned two frames of images to be estimated can be the previous frame of the image to be estimated and the next frame of the image to be estimated, which are divided in order of sampling time. Accordingly, the first depth map of the above-mentioned image to be estimated includes the first depth map of the previous frame of the image to be estimated and the first depth map of the next frame of the image to be estimated.

[0066] The above-mentioned use of a preset initial depth estimation model to perform depth estimation on two frames of images to be estimated to obtain the first depth map of the image to be estimated can be performed by using the depth encoding network of the preset initial depth estimation model to obtain depth features of the image to be estimated, and then using the depth decoding network of the preset initial depth estimation model to extract features related to depth estimation on the obtained depth features to obtain an inverse depth map of the image to be estimated. Finally, the inverse depth map is taken to obtain the first depth map of the image to be estimated. The inverse depth map of the image to be estimated is the inverse of the depth map of the image to be estimated. The first depth map of the image to be estimated can be obtained by taking the inverse of the inverse depth map. It should be noted that the size of the first depth map of the image to be estimated is related to the downsampling multiple in the depth encoding network of the initial depth estimation model and the upsampling multiple in the depth decoding network of the initial depth estimation model. If the downsampling multiple in the depth encoding network of the initial depth estimation model is equal to the upsampling multiple in the depth decoding network of the initial depth estimation model, the size of the first depth map of the image to be estimated is the same as the size of the image to be estimated. If the size of the first depth map of the image to be estimated is larger than the size of the image to be estimated, the first depth map of the image to be estimated can be scaled so that the size of the first depth map of the image to be estimated is consistent with the size of the image to be estimated. Using the method in this step, the first depth map of the image to be estimated can be obtained, facilitating the subsequent calculation of the projection error using the first depth map of the image to be estimated.

[0067] In step S220 , a preset initial point cloud estimation model is used to perform point cloud estimation on two frames of millimeter wave point clouds to be estimated corresponding to the two frames of images to be estimated, and a dynamic point set and a first pose transformation result of the millimeter wave point clouds to be estimated are obtained.

[0068] In which, the above-mentioned initial point cloud estimation model may include a scene flow prediction network and a pose estimation network. In which, the scene flow prediction network may include a feature extraction layer for obtaining the scene flow of the millimeter wave point cloud to be estimated. The pose estimation network may reuse the feature extraction layer of the scene flow prediction network, and replace all network layers of the scene flow prediction network of the initial point cloud estimation model starting from the upsampling convolution layer with N linear layers, and the output of the last linear layer is a matrix with one row and six columns. In which, the number N and the input and output of each linear layer can be preset. The pose estimation network can be used to estimate the pose transformation result between two frames of millimeter wave point clouds to be estimated. The above-mentioned method uses the preset initial point cloud estimation model to perform point cloud estimation on the two frames of millimeter wave point clouds to be estimated corresponding to the two frames of pictures to be estimated, and obtains the dynamic point set and the first pose transformation result of the millimeter wave point cloud to be estimated. It can be that the scene flow prediction network of the preset initial point cloud estimation model is used to obtain the scene flow of the millimeter wave point cloud to be estimated. Then, according to the preset dynamic point screening conditions, the dynamic points in the scene flow that meet the translation deviation greater than or equal to one times the variance of the average translation offset are screened out to obtain the dynamic point set of the millimeter wave point cloud to be estimated. The pose estimation network of the preset initial point cloud estimation model can be used to obtain a one-row and six-column matrix of the millimeter wave point cloud to be estimated, and then, using the preset rotation formula, the one-row and six-column matrix is ​​converted into a three-row and four-column matrix to obtain the first pose transformation result of the millimeter wave point cloud to be estimated. The above-mentioned preset rotation formula can be the Rodrigues rotation formula. Among them, the one-row and six-column matrix of the millimeter wave point cloud to be estimated can be a matrix in which the six columns respectively represent the three angle values ​​of Pitch, Yaw, and Roll and the translation amounts in the directions represented by the three coordinate axes of X, Y, and Z. Pitch refers to the pitch angle, which is the angle between the head and the horizontal plane when looking up or down, and can also be interpreted as the angle of rotation along the Y axis of the local coordinate system (the coordinate system with the X axis forward); Yaw refers to the yaw angle, which is the angle of rotation along the Z axis of the world coordinate system; Roll refers to the roll angle, which is the angle between the left and right tilt and the horizontal plane, and can also be interpreted as the angle of rotation along the X axis of the local coordinate system (the coordinate system with the X axis forward). For example, the above matrix A with one row and six columns can be expressed as: A = [Pitch Yaw Roll XYZ]

[0069] In the above three-row and four-column matrix, the three rows can refer to the three angle values ​​of Pitch, Yaw, and Roll, and the four columns can refer to the translation amounts in the directions represented by the three coordinate axes X, Y, and Z, as well as the value of the radial velocity V. For example, the above three-row and four-column matrix B can be expressed as:

[0070] The first pose transformation result can be a transformation matrix that converts a matrix with one row and six columns into a matrix with three rows and four columns. This step allows for the acquisition of a dynamic point set and the first pose transformation result for the millimeter-wave point cloud to be estimated, facilitating the subsequent calculation of pose estimation errors and depth errors of moving objects using the dynamic point set and the first pose transformation result.

[0071] Step S230, based on the first pose transformation result, calculate the camera's extrinsic parameter transformation value; based on the camera's extrinsic parameter transformation value and the camera's intrinsic parameter value, project the first depth map of the previous frame of the picture to be estimated to the viewing angle of the next frame of the picture to be estimated to obtain the second depth map of the next frame of the picture to be estimated; according to a preset projection error calculation method, calculate the projection error between the first depth map of the next frame of the picture to be estimated and the second depth map of the next frame of the picture to be estimated.

[0072] The above calculation of the camera's extrinsic parameter transformation value based on the first pose transformation result can be based on the first pose transformation result and the preset extrinsic parameter value of the millimeter wave radar to the camera to obtain the camera's extrinsic parameter transformation value corresponding to the two frames of the estimated image. For example, the camera's extrinsic parameter transformation value TC corresponding to the image to be estimated at time T-1 to the image to be estimated at time T is T-1 →T The specific calculation formula is: TC T-1→T =T R→C -1 TR T-1→T T R→C

[0073] Among them, T R→C represents the external parameter value of the millimeter wave radar to the camera, TR T-1→T Indicates the first pose transformation result from the millimeter wave point cloud to be estimated at time T-1 to the millimeter wave point cloud to be estimated at time T.

[0074] The second depth map of the next frame to be estimated can be obtained by projecting the first depth map of the previous frame to be estimated onto the perspective of the next frame to be estimated based on the camera's extrinsic parameter transformation value and the camera's intrinsic parameter value, and obtaining the second depth map of the next frame to be estimated based on the projection result. For example, it can be the camera's extrinsic parameter transformation value TC corresponding to the picture to be estimated at time T-1 to the picture to be estimated at time T. T-1→T And the intrinsic parameter value of the camera, the first depth map D of the image to be estimated at time T-1 T-1 The perspective of the image to be estimated at time T is projected, and based on the projection result, the second depth map D of the image to be estimated at time T is obtained T-1→T The second depth map D of the image to be estimated at time T T-1→T The calculation formula is as follows: T-1→T =D T-1 TC T-1→T-1

[0075] The above-mentioned projection error calculation method based on the preset projection error calculation method calculates the projection error of the first depth map of the next frame to be estimated and the second depth map of the next frame to be estimated, which can be calculated by using the SSIM (Structure Similarity Index Measure) loss function to calculate the projection error loss of the first depth map of the next frame to be estimated and the second depth map of the next frame to be estimated. For example, the SSIM loss function is used to calculate the first depth map D of the picture to be estimated at time T. T And the second depth map D of the image to be estimated at time T T-1→T The calculation formula of the projection error L1 is:

[0076] in, are pre-set parameters.

[0077] In step S240, a preset estimation algorithm is used to estimate the overall pose transformation of the two frames of millimeter-wave point clouds to be estimated, and a second pose transformation result of the overall pose transformation of the millimeter-wave point clouds to be estimated is obtained; based on the first pose transformation result and the second pose transformation result, according to the preset pose estimation error calculation method, the pose estimation error of the first pose transformation result and the second pose transformation result is obtained.

[0078] In this step, the above-mentioned preset estimation algorithm is used to estimate the overall posture transformation of the two frames of millimeter-wave point clouds to be estimated, and the second posture transformation result of the overall posture transformation of the millimeter-wave point cloud to be estimated is obtained. It can be that the ICP (Iterative Closest Point, iterative closest point cloud matching algorithm) algorithm is used to estimate the overall posture transformation of the two frames of millimeter-wave point clouds to be estimated, and the second posture transformation result of the overall posture transformation of the millimeter-wave point cloud to be estimated is obtained. Among them, according to the preset posture estimation error calculation method, the posture estimation error of the first posture transformation result and the second posture transformation result is obtained. It can be calculated according to the preset posture estimation error calculation formula. The calculation formula for the posture estimation error L2 of the above-mentioned preset first posture transformation result and the second posture transformation result is:

[0079] Among them, TR1 represents the first pose transformation result, and TR2 represents the second pose transformation result.

[0080] Step S250 : Based on the first depth map and the dynamic point set, and according to a preset method for calculating the depth error of a moving object, calculate the depth error of the moving object in the two frames of images to be estimated.

[0081] The above-mentioned method of calculating the depth error of the moving object according to the preset depth error calculation method can be used to calculate the depth error of the moving object in the two frames of the picture to be estimated. The depth error of the moving object in the two frames of the picture to be estimated can be calculated based on the probability that any point pair p and q in the two frames of millimeter wave point clouds corresponding to the two frames of the picture to be estimated come from the same obstacle, whether the three-dimensional vector formed at point q is consistent with the overall translation method of the two frames of millimeter wave point clouds before and after, and the depth value of the first depth map of the two frames of the picture to be estimated at the position corresponding to the two frames of millimeter wave point clouds where the point pair p and q are located. The depth error of the moving object in the two frames of the picture to be estimated is calculated by the formula:

[0082] Among them, RD T-1 Represents the dynamic point set at time T-1, RD T represents the dynamic point set at time T, p and q represent RD T-1 and RD T For any pair of points in , Sim(p,q) represents the probability that p and q come from the same obstacle, Loc q represents the three-dimensional space coordinates of point q with the millimeter wave as the origin, TR T-1→T Denotes the first pose transformation result of the millimeter wave point cloud to be estimated from time T-1 to time T, D T (p) represents the depth value of point p in the first depth map of the image to be estimated at time T, D T-1 (q) represents the depth value of point q in the first depth map of the image to be estimated at time T-1.

[0083] The calculation formula of Sim(p,q) is as follows:

[0084] Among them, Loc p Indicates the three-dimensional space coordinates of p, Loc q represents the three-dimensional space coordinates of q, V p and V q is a scalar at point P and point q, representing the radial velocity at point p and point q.

[0085] Among them, π(cos(Loc q , TR T-1→T [:, 3])) is the indicator function. When the three-dimensional vector formed at point q is consistent with the overall translation direction of the millimeter-wave point cloud of the previous and next two frames, the value of the indicator function is 1, otherwise the value of the indicator function is 0.

[0086] Among them, π(D T (p)<D T-1(q)) is an indicator function, indicating that when the depth value of p in the first depth map of the image to be estimated at time T is less than the depth value of q in the first depth map of the image to be estimated at time T-1, the value of the indicator function is 1; otherwise, the value of the indicator function is 0.

[0087] Step S260, according to the projection error of the first depth map of the next frame of the picture to be estimated and the second depth map of the next frame of the picture to be estimated, the pose estimation error of the first pose transformation result and the second pose transformation result, and the depth error of the moving object in the two frames of the picture to be estimated, obtain the overall training loss of the picture to be estimated, and use the overall training loss to train the initial depth estimation model and the initial point cloud estimation model until the initial depth estimation model and the initial point cloud estimation model converge, so as to obtain a complete depth estimation model for monocular image depth estimation.

[0088] In this step, based on the projection error of the first depth map of the next frame to be estimated and the second depth map of the next frame to be estimated, the pose estimation error of the first pose transformation result and the second pose transformation result, and the depth error of the moving object in the two frames to be estimated, the calculation formula for the overall training loss L of the picture to be estimated is:

[0089] L=L1+β·L2+(1-β)·L3

[0090] in, ep represents the current training epoch, and Max_epoch represents the maximum training epoch.

[0091] The convergence condition of the above-mentioned initial depth estimation model and initial point cloud estimation model can be that the overall training loss L of the image to be estimated reaches a preset threshold, or that the number of training times of the initial depth estimation model and initial point cloud estimation model reaches a preset threshold.

[0092] Step S270 : performing monocular image depth estimation on the image to be estimated based on the complete depth estimation model.

[0093] In the above steps S210 to S270, a first depth map of the image to be estimated is obtained by using a preset initial depth estimation model, and a dynamic point set and a first pose transformation result of the millimeter-wave point cloud to be estimated are obtained by using a preset initial point cloud estimation model. Then, the first depth map of the next frame of the image to be estimated is calculated using a preset algorithm to obtain a second depth map of the next frame of the image to be estimated, and the projection error between the first depth map of the next frame of the image to be estimated and the second depth map of the next frame of the image to be estimated is calculated. The pose transformation calculation of the two frames of millimeter-wave point clouds to be estimated is performed using a preset algorithm to obtain a second pose transformation result, and the pose estimation error of the first pose transformation result and the second pose transformation result is calculated. Finally, the depth error of the moving object in the two frames of the image to be estimated is calculated. The projection error, pose estimation error and depth error of the moving object are then fed back to the overall training loss for training the initial depth estimation model. This approach can take into account the moving objects in the image and obtain an accurate and complete depth estimation model, thereby ensuring the stability of the depth estimation results of the image achieved using the complete depth estimation model, solving the problem that related technologies cannot guarantee the stability of the depth estimation results of the image.

[0094] In one embodiment, in step S210, before performing depth estimation on two frames of images to be estimated using a preset initial depth estimation model to obtain a first depth map of the images to be estimated, the following steps are included:

[0095] Step S202 : performing a mean subtraction and variance division operation on the two frames of original pictures to be estimated to generate two frames of first pictures.

[0096] In this step, the subtraction of the mean and the division of the variance operation on the two frames of the original image to be estimated may be performed on the two frames of the original image to be estimated according to channels.

[0097] Step S204 : Using a preset scaling method, the two frames of the first image are scaled to a preset size to obtain two scaled frames of images to be estimated.

[0098] The preset scaling methods mentioned above may be methods capable of scaling an image, such as nearest neighbor scaling, bilinear scaling, and bicubic scaling. It should be noted that scaling the first image may be performed using one or more of the aforementioned methods, or other methods that can be used for image scaling, and are not specifically limited herein.

[0099] In the above steps S202 to S204, two frames of first images are generated by performing mean subtraction and variance division operations on the two frames of original images to be estimated, and then the two frames of first images are scaled so that the first images are scaled to a preset size to obtain the images to be estimated. This can improve the efficiency of depth estimation of the images to be estimated using the initial depth estimation model.

[0100] The present embodiment is described and illustrated below through optional embodiments.

[0101] FIG3 is a flow chart of a monocular image depth estimation method provided by an optional embodiment of the present application. As shown in FIG3 , the monocular image depth estimation method includes the following steps S310 to S390.

[0102] Step S310, performing a mean subtraction and variance division operation on the two frames of original images to be estimated to generate two frames of first images;

[0103] Step S320: scaling the two first images to a preset size using a preset scaling method to obtain two scaled images to be estimated.

[0104] Step S330, using a preset initial depth estimation model, performing depth estimation on the two frames of the image to be estimated to obtain a first depth map of the image to be estimated;

[0105] Step S340: Using a preset initial point cloud estimation model, perform point cloud estimation on two frames of millimeter-wave point clouds to be estimated corresponding to the two frames of images to be estimated, and obtain a dynamic point set and a first pose transformation result of the millimeter-wave point clouds to be estimated;

[0106] Step S350: obtaining a second depth map of the next frame of the image to be estimated based on the first pose transformation result and the intrinsic parameter value of the camera, and then calculating the projection error between the first depth map of the next frame of the image to be estimated and the second depth map of the next frame of the image to be estimated;

[0107] Step S360, performing overall pose transformation estimation on the two frames of millimeter-wave point clouds to be estimated, obtaining a second pose transformation result of the overall pose transformation of the millimeter-wave point clouds to be estimated, and then calculating a pose estimation error between the first pose transformation result and the second pose transformation result based on the first pose transformation result and the second pose transformation result;

[0108] Step S370: Calculate the depth error of the moving object in the two frames of the image to be estimated based on the first depth map and the dynamic point set according to a preset method for calculating the depth error of the moving object;

[0109] Step S380: Calculate the overall training loss of the image to be estimated based on the projection error, the pose estimation error, and the depth error of the moving object, and use the overall training loss to train the initial depth estimation model and the initial point cloud estimation model until the initial depth estimation model and the initial point cloud estimation model converge, thereby obtaining a complete depth estimation model for monocular image depth estimation;

[0110] Step S390: performing monocular image depth estimation on the image to be estimated based on the complete depth estimation model.

[0111] In the above steps S310 to S390, two frames of first images are generated by performing mean subtraction and variance division operations on the two frames of original images to be estimated, and then the two frames of first images are scaled to a preset size using a preset scaling method to obtain the scaled two frames of images to be estimated. Then, a first depth map of the image to be estimated is obtained by using a preset initial depth estimation model, and a dynamic point set and a first pose transformation result of the millimeter wave point cloud to be estimated are obtained by using a preset initial point cloud estimation model. Then, a preset algorithm is used to calculate the first depth map of the next frame of the image to be estimated to obtain a second depth map of the next frame of the image to be estimated, and the projection error between the first depth map of the next frame of the image to be estimated and the second depth map of the next frame of the image to be estimated is calculated. The two frames of millimeter wave point clouds to be estimated are subjected to pose transformation calculation using a preset algorithm to obtain a second pose transformation result, and the pose estimation error between the first pose transformation result and the second pose transformation result is calculated. Finally, the depth error of the moving object in the two frames of the image to be estimated is calculated. The projection error, pose estimation error and depth error of the moving object are then fed back to the overall training loss for training the initial depth estimation model. This approach can take into account the moving objects in the image and obtain an accurate and complete depth estimation model, thereby ensuring the stability of the depth estimation results of the image achieved using the complete depth estimation model, solving the problem that related technologies cannot guarantee the stability of the depth estimation results of the image.

[0112] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0113] Based on the same inventive concept, a monocular image depth estimation device 400 is also provided in this embodiment, which is used to implement the above-mentioned embodiments and optional implementation methods. The details that have been explained will not be repeated here. The terms "module", "unit", "sub-unit", etc. used below can be a combination of software and / or hardware that can implement predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and conceived.

[0114] In one embodiment, FIG4 is a structural block diagram of a monocular image depth estimation device 400 provided in an embodiment of the present application. As shown in FIG4 , the monocular image depth estimation device 400 includes:

[0115] The depth estimation module 41 is used to perform depth estimation on two frames of images to be estimated using a preset initial depth estimation model to obtain a first depth map of the images to be estimated; the first depth map includes the first depth map of the previous frame and the first depth map of the next frame.

[0116] The point cloud estimation module 42 is used to use a preset initial point cloud estimation model to perform point cloud estimation on two frames of millimeter wave point clouds to be estimated corresponding to the two frames of images to be estimated, and obtain a dynamic point set and a first pose transformation result of the millimeter wave point clouds to be estimated.

[0117] The first calculation module 43 is used to calculate the camera's extrinsic parameter transformation value based on the first pose transformation result; based on the camera's extrinsic parameter transformation value and the camera's intrinsic parameter value, project the first depth map of the previous frame of the picture to be estimated to the perspective of the next frame of the picture to be estimated to obtain the second depth map of the next frame of the picture to be estimated; according to a preset projection error calculation method, calculate the projection error between the first depth map of the next frame of the picture to be estimated and the second depth map of the next frame of the picture to be estimated.

[0118] The second calculation module 44 is used to use a preset estimation algorithm to perform overall pose transformation estimation on two frames of millimeter-wave point clouds to be estimated, and obtain a second pose transformation result of the overall pose transformation of the millimeter-wave point clouds to be estimated; based on the first pose transformation result and the second pose transformation result, according to the preset pose estimation error calculation method, obtain the pose estimation error of the first pose transformation result and the second pose transformation result.

[0119] The third calculation module 45 is configured to calculate the depth error of the moving object in the two frames of the to-be-estimated pictures based on the first depth map and the dynamic point set and according to a preset method for calculating the depth error of the moving object.

[0120] The training module 46 is used to obtain the overall training loss of the image to be estimated based on the projection error of the first depth map of the next frame of the image to be estimated and the second depth map of the next frame of the image to be estimated, the pose estimation error of the first pose transformation result and the second pose transformation result, and the depth error of the moving object in the two frames of the image to be estimated, and use the overall training loss to train the initial depth estimation model and the initial point cloud estimation model until the initial depth estimation model and the initial point cloud estimation model converge, thereby obtaining a complete depth estimation model for monocular image depth estimation.

[0121] And an estimation module 47 is used to perform monocular image depth estimation on the image to be estimated based on the complete depth estimation model.

[0122] The above-mentioned monocular image depth estimation device 400 obtains the first depth map of the image to be estimated by using a preset initial depth estimation model, and obtains the dynamic point set and the first pose transformation result of the millimeter wave point cloud to be estimated by using a preset initial point cloud estimation model. Then, a preset algorithm is used to calculate the second depth map of the next frame of the image to be estimated to obtain the second depth map of the next frame of the image to be estimated, and the projection error between the first depth map of the next frame of the image to be estimated and the second depth map of the next frame of the image to be estimated is calculated. The pose transformation calculation is performed on the two frames of millimeter wave point clouds to be estimated by using a preset algorithm to obtain the second pose transformation result, and the pose estimation error of the first pose transformation result and the second pose transformation result is calculated. Finally, the depth error of the moving object in the two frames of the image to be estimated is calculated. The projection error, pose estimation error and depth error of the moving object are then fed back to the overall training loss for training the initial depth estimation model. This approach can take into account the moving objects in the image and obtain an accurate and complete depth estimation model, thereby ensuring the stability of the depth estimation results of the image achieved using the complete depth estimation model, solving the problem that related technologies cannot guarantee the stability of the depth estimation results of the image.

[0123] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0124] In one embodiment, FIG5 is a block diagram of a computer device according to some embodiments. As shown in FIG5 , a computer device 500 is provided, including a memory 104 and a processor 102. The memory 104 stores a computer program, and when the processor 102 executes the computer program, it implements any of the monocular image depth estimation methods described in the above embodiments.

[0125] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0126] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0127] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0128] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A monocular image depth estimation method, characterized in that: The method comprises: Using a preset initial depth estimation model, depth estimation is performed on two frames of pictures to be estimated to obtain a first depth map of the pictures to be estimated; the first depth map of the pictures to be estimated includes the first depth map of the previous frame of the picture to be estimated and the first depth map of the next frame of the picture to be estimated; Using a preset initial point cloud estimation model, point cloud estimation is performed on two frames of millimeter-wave point clouds to be estimated corresponding to the two frames of the pictures to be estimated, so as to obtain a dynamic point set and a first pose transformation result of the millimeter-wave point clouds to be estimated; Based on the first pose transformation result, calculating the extrinsic parameter transformation value of the camera; based on the extrinsic parameter transformation value of the camera and the intrinsic parameter value of the camera, projecting the first depth map of the previous frame of the picture to be estimated to the viewing angle of the next frame of the picture to be estimated, to obtain the second depth map of the next frame of the picture to be estimated; calculating the projection error between the first depth map of the next frame of the picture to be estimated and the second depth map of the next frame of the picture to be estimated according to a preset projection error calculation method; Using a preset estimation algorithm, the overall posture transformation of the two frames of the millimeter-wave point cloud to be estimated is estimated to obtain a second posture transformation result of the overall posture transformation of the millimeter-wave point cloud to be estimated; based on the first posture transformation result and the second posture transformation result, according to a preset posture estimation error calculation method, the posture estimation error of the first posture transformation result and the second posture transformation result is obtained; Based on the first depth map and the dynamic point set, and according to a preset method for calculating the depth error of a moving object, calculate the depth error of the moving object in the two frames of the pictures to be estimated; According to the projection errors of the first depth map of the next frame of the picture to be estimated and the second depth map of the next frame of the picture to be estimated, the pose estimation errors of the first pose transformation result and the second pose transformation result, and the depth errors of the moving objects in the two frames of the pictures to be estimated, the overall training loss of the picture to be estimated is obtained, and the initial depth estimation model and the initial point cloud estimation model are trained by using the overall training loss until the initial depth estimation model and the initial point cloud estimation model converge, so as to obtain a complete depth estimation model for monocular image depth estimation; Based on the complete depth estimation model, monocular image depth estimation is performed on the picture to be estimated.

2. The monocular image depth estimation method according to claim 1, wherein: Before performing depth estimation on two frames of pictures to be estimated by using a preset initial depth estimation model to obtain a first depth map of the pictures to be estimated, the method includes: Perform mean subtraction and variance division operations on the two frames of original images to be estimated to generate two frames of first images; The two frames of the first pictures are scaled to a preset size using a preset scaling method to obtain the scaled two frames of the to-be-estimated pictures.

3. The monocular image depth estimation method according to claim 1, wherein: The method of using a preset initial depth estimation model to perform depth estimation on two frames of images to be estimated to obtain a first depth map of the images to be estimated includes: Acquire the depth feature of the image to be estimated by using the preset depth coding network of the initial depth estimation model; Using the preset depth decoding network of the initial depth estimation model, extracting features related to depth estimation from the acquired depth features to obtain an inverse depth map of the image to be estimated; The inverse depth map is processed by taking the inverse, so as to obtain a first depth map of the picture to be estimated.

4. The monocular image depth estimation method according to claim 1, wherein: The method of using a preset initial point cloud estimation model to perform point cloud estimation on two frames of millimeter wave point clouds to be estimated corresponding to the two frames of pictures to be estimated, and obtaining a dynamic point set and a first pose transformation result of the millimeter wave point clouds to be estimated, includes: Acquire the scene flow of the millimeter-wave point cloud to be estimated by using the preset scene flow prediction network of the initial point cloud estimation model; According to the preset dynamic point screening condition, the dynamic points in the scene stream that satisfy the translation deviation greater than or equal to one times the variance of the average translation offset are screened out to obtain the dynamic point set of the millimeter wave point cloud to be estimated; Using the preset pose estimation network of the initial point cloud estimation model, obtaining a matrix of one row and six columns of the millimeter wave point cloud to be estimated; The matrix of one row and six columns is converted into a matrix of three rows and four columns by using a preset rotation formula, so as to obtain the first posture transformation result of the millimeter wave point cloud to be estimated.

5. The monocular image depth estimation method according to claim 1, wherein: The method of calculating the external parameter transformation value of the camera based on the first pose transformation result; and projecting the first depth map of the previous frame of the picture to be estimated to the viewing angle of the next frame of the picture to be estimated based on the external parameter transformation value of the camera and the internal parameter value of the camera to obtain the second depth map of the next frame of the picture to be estimated, including: Based on the first posture transformation result and the preset extrinsic parameter value of the millimeter wave radar to the camera, obtain the extrinsic parameter transformation value of the camera corresponding to the two frames of the pictures to be estimated; Based on the extrinsic parameter transformation value of the camera and the intrinsic parameter value of the camera, the first depth map of the previous frame of the picture to be estimated is projected to the viewing angle of the next frame of the picture to be estimated, and based on the projection result, the second depth map of the next frame of the picture to be estimated is obtained.

6. The monocular image depth estimation method according to claim 1, wherein: The calculation formula of the projection error L1 between the first depth map of the next frame to be estimated picture and the second depth map of the next frame to be estimated picture is: Among them, D T Denotes the first depth map of the image to be estimated at time T, D T-1→T It represents the first depth map of the image to be estimated at time T-1 projected to the viewing angle of the image to be estimated at time T, and the second depth map of the image to be estimated at time T is obtained. SSIM means using the SSIM loss function to calculate the projection error loss of the first depth map of the image to be estimated at time T and the second depth map of the image to be estimated at time T. are pre-set parameters.

7. The monocular image depth estimation method according to claim 1, wherein: The method of using a preset estimation algorithm to estimate the overall posture transformation of the two frames of the to-be-estimated millimeter-wave point clouds to obtain a second posture transformation result of the overall posture transformation of the millimeter-wave point clouds includes: An iterative closest point cloud matching algorithm is used to estimate the overall pose transformation of the two frames of the millimeter-wave point cloud to be estimated, and a second pose transformation result of the overall pose transformation of the millimeter-wave point cloud to be estimated is obtained.

8. The monocular image depth estimation method according to claim 1, wherein: The calculation formula of the pose estimation error L2 between the first pose transformation result and the second pose transformation result is: Among them, TR1 represents the first posture transformation result, and TR2 represents the second posture transformation result.

9. The monocular image depth estimation method according to claim 1, wherein: The calculation formula of the depth error L3 of the moving object in the two frames of the to-be-estimated pictures is: Among them, RD T-1 Represents the dynamic point set at time T-1, RD T represents the dynamic point set at time T, p and q represent RD T-1 and RD T For any pair of points in , Sim(p, q) represents the probability that p and q come from the same obstacle, π(cos(Loc q , TR T-1→T [:,3])) is the indicator function, Loc q represents the three-dimensional space coordinates of point q, TR T-1→T represents the first pose transformation result of the millimeter wave point cloud to be estimated from time T-1 to time T, π(D T (p)<D T-1 (q)) is the indicator function, D T (p) represents the depth value of p in the first depth map of the image to be estimated at time T, D T-1 (q) represents the depth value of q in the first depth map of the image to be estimated at time T-1.

10. The monocular image depth estimation method according to claim 1, wherein: The overall training loss formula L of the image to be estimated is: L = L1 + β·L2 + (1-β)·L3 in, ep represents the current training round, and Max_epoch represents the maximum training round.

11. A monocular image depth estimation device, characterized in that: The device comprises: A depth estimation module is used to perform depth estimation on two frames of images to be estimated using a preset initial depth estimation model to obtain a first depth map of the images to be estimated; the first depth map of the images to be estimated includes the first depth map of the previous frame of the image to be estimated and the first depth map of the next frame of the image to be estimated; A point cloud estimation module, used to perform point cloud estimation on two frames of millimeter-wave point clouds to be estimated corresponding to the two frames of pictures to be estimated, using a preset initial point cloud estimation model, to obtain a dynamic point set and a first pose transformation result of the millimeter-wave point clouds to be estimated; A first calculation module is used to calculate the external parameter transformation value of the camera based on the first posture transformation result; based on the external parameter transformation value of the camera and the internal parameter value of the camera, project the first depth map of the previous frame of the picture to be estimated to the viewing angle of the next frame of the picture to be estimated, and obtain the second depth map of the next frame of the picture to be estimated; according to a preset projection error calculation method, calculate the projection error between the first depth map of the next frame of the picture to be estimated and the second depth map of the next frame of the picture to be estimated; A second calculation module is used to use a preset estimation algorithm to perform overall posture transformation estimation on the two frames of the millimeter-wave point cloud to be estimated, and obtain a second posture transformation result of the overall posture transformation of the millimeter-wave point cloud to be estimated; based on the first posture transformation result and the second posture transformation result, according to a preset posture estimation error calculation method, obtain the posture estimation error of the first posture transformation result and the second posture transformation result; A third calculation module is used to calculate the depth error of the moving object in the two frames of the to-be-estimated pictures based on the first depth map and the dynamic point set and according to a preset depth error calculation method of the moving object; A training module, configured to obtain an overall training loss of the picture to be estimated based on a projection error of a first depth map of the next frame of the picture to be estimated and a second depth map of the next frame of the picture to be estimated, a pose estimation error of the first pose transformation result and the second pose transformation result, and a depth error of the moving object in the two frames of the pictures to be estimated, and train the initial depth estimation model and the initial point cloud estimation model using the overall training loss until the initial depth estimation model and the initial point cloud estimation model converge, thereby obtaining a complete depth estimation model for monocular image depth estimation; And an estimation module, which is used to perform monocular image depth estimation on the picture to be estimated based on the complete depth estimation model.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the monocular image depth estimation method described in any one of claims 1 to 10 are implemented.

Images