Monocular image depth estimation method and system
By constructing a deep network and combining photometric consistency, edge smoothing and confidence depth loss function, a self-supervised depth estimation method is proposed to solve the problem of insufficient depth estimation accuracy under complex occlusion of urban roads and improve the estimation accuracy of low texture areas.
Patent Information
- Application Number
- CN202511084924.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-04
- Publication Date
- 2025-11-18
AI Technical Summary
Existing depth estimation methods based on aerial imagery are ill-equipped to handle complex 3D occlusion problems in urban roads, especially in situations such as multi-layered traffic structures on elevated bridges and tree canopy occlusion. Traditional methods suffer from discontinuous depths at different semantic boundaries, and methods based on geometric features and deep learning lack sufficient estimation accuracy in low-texture regions.
A deep network is constructed and a self-supervised depth estimation network is trained using photometric consistency, edge smoothing, and confidence depth loss functions. The depth estimation method is optimized using a pose network and a semantic segmentation model to enhance the estimation accuracy of low-texture small targets and texture artifact regions.
It improves the estimation accuracy of regions with discontinuous depths at different semantic boundaries, enhances the depth estimation capability of low-texture small targets and texture artifact regions, and solves the failure problem of traditional methods.
Smart Images
Figure CN120976286A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of monocular image depth estimation, and specifically relates to a monocular image depth estimation method and system. Background Technology
[0002] The goal of monocular image depth estimation methods is to address the failure problem of depth discontinuities at different semantic boundaries in traditional methods and to optimize prediction accuracy. In typical scenarios such as emergency rescue and smart city construction, the wide-area observation information provided by UAVs offers crucial support for navigation decisions of ground vehicles.
[0003] However, existing depth estimation methods based on aerial imagery struggle to address the prevalent 3D occlusion problems in urban roads—such as vertical spatial occlusion caused by multi-layered traffic structures on elevated highways and tree canopy occlusion. Previous patents have focused on geometric features or deep learning. Geometric feature-based methods infer depth using geometric cues in the image or hand-designed features, but they rely on strong assumptions, have poor generalization ability, fail to handle textured or repetitive areas (such as white walls or the sky), and cannot handle complex scenes. Deep learning-based methods produce poor depth estimates for some images containing low-texture small targets or texture artifacts (such as bridge shadows and road surfaces underneath), especially for long-distance depth information, which is highly unreliable.
[0004] Therefore, there is an urgent need for a technical solution to overcome or mitigate at least one of the aforementioned defects in the existing technology. Summary of the Invention
[0005] The purpose of this application is to provide a monocular image depth estimation method and system to solve the failure problem of traditional methods due to depth discontinuity at different semantic boundaries.
[0006] The technical solution of this application is:
[0007] The first aspect of this application provides a method for monocular image depth estimation, including:
[0008] S10. Construct a deep network and obtain a depth map of the image through the deep network;
[0009] S20. Construct a loss function based on the depth map, the loss function including a photometric consistency loss function, an edge smoothness loss function, and a confidence depth loss function;
[0010] S30. Train a self-supervised depth estimation network based on the loss function to achieve monocular image depth estimation.
[0011] In at least one embodiment of this application, in step S20, constructing a loss function based on the depth map includes:
[0012] S21. Construct a pose network to obtain the relative camera pose of two adjacent frames of images, and construct a photometric consistency loss function based on the depth map and the relative camera pose.
[0013] S22. Construct a semantic segmentation model, divide the image into regions using the semantic segmentation model to obtain semantic segmentation regions of different semantic categories, obtain the set of pixels corresponding to the semantic categories from the semantic segmentation regions, and construct an edge smoothness loss function based on the depth map and the set of pixels.
[0014] S23. Obtain a pre-trained monocular depth estimation model, obtain a prior depth map of the image through the monocular depth estimation model, and sample the prior depth map and the depth map to obtain sampling point pairs, and construct a confidence depth loss function based on the sampling point pairs.
[0015] S24. The loss function is obtained by adding the photometric consistency loss function, the edge smoothness loss function and the confidence depth loss function.
[0016] In at least one embodiment of this application, in step S21, a pose network is constructed to obtain the relative camera pose of two adjacent frames. A photometric consistency loss function is then constructed based on the depth map and the relative camera pose, including:
[0017] Construct a pose network to obtain the relative camera pose of two adjacent frames;
[0018] A synthetic target view image is obtained based on the depth map and the relative camera pose;
[0019] Based on the difference between the synthesized target view image and the measured target view image, a photometric consistency loss function is constructed:
[0020]
[0021] in, For photometric consistency loss, SSIM is used to calculate structural similarity. To synthesize the target view image, I t The image represents the measured target view, α represents the balancing weight, and ||*|| represents the computational norm.
[0022] In at least one embodiment of this application, in step S22, constructing an edge smoothness loss function based on the depth map and the set of pixels includes:
[0023] The depth map gradient is calculated based on the depth map.
[0024] The gradient of the depth map is normalized;
[0025]
[0026] Where, d * Here, d represents the normalized depth map gradient. This represents the mean of the depth map gradient.
[0027] Based on the normalized depth map gradient, construct the edge smoothness loss function:
[0028]
[0029] in, For edge smoothness loss, For the horizontal gradient operator, Here, E(p) is the vertical gradient operator, E(p) is the intensity value of the image at pixel p, |*| is used to calculate the absolute value, ||*|| is used to calculate the norm, Ω is the set of pixels, and N is the number of pixels.
[0030] The set of pixels Ω for semantic category k k ={(i,j)|S ij For each pixel (i,j) in {k}, calculate the edge smoothness loss based on the depth map gradient and the image gradient weights respectively:
[0031]
[0032] in, For the edge smoothness loss of semantic category k, N k The number of pixels for semantic category k;
[0033] The edge smoothness loss for all semantic categories is summed to obtain the edge smoothness loss function for the entire image based on semantic segmentation:
[0034]
[0035] in, The edge smoothness loss is based on semantic segmentation.
[0036] In at least one embodiment of this application, in step S23, a pre-trained monocular depth estimation model is obtained, a prior depth map of the image is obtained through the monocular depth estimation model, and the prior depth map and the depth map are sampled to obtain sampling point pairs, including:
[0037] Obtain a pre-trained monocular depth estimation model, and use the monocular depth estimation model to obtain the prior depth map of the image;
[0038] Random sampling is performed on the prior depth map region corresponding to the semantic segmentation region of semantic category k in the image to obtain a first sampling point, and the predicted depth value of the first sampling point is...
[0039] The same location corresponding to the first sampling point in the depth map is sampled to obtain a second sampling point, and the predicted depth value of the second sampling point is...
[0040] The predicted depth value is obtained based on the first sampling point and the second sampling point. The sampling point pair.
[0041] In at least one embodiment of this application, in step S23, constructing a confidence depth loss function based on the sampling point pair includes:
[0042] The confidence depth loss function is:
[0043]
[0044] in, For confidence depth loss function, The depth order labels are obtained by inducing the prior depth map.
[0045] Determine depth order labels based on prior depth ratios:
[0046]
[0047] in, Let τ be the prior depth of the sampling point pair, and τ be the adjustment coefficient.
[0048] The confidence depth loss function corresponding to semantic category k is:
[0049]
[0050] in, Let k be the confidence depth loss function corresponding to semantic category k. The number of sample point pairs for semantic category k;
[0051] By assigning corresponding weight coefficients to different semantic categories k, the confidence depth loss function for semantic constraints is obtained:
[0052]
[0053] Among them, L CDS The confidence depth loss function for semantic constraints. represents the weight coefficient corresponding to semantic category k.
[0054] In at least one embodiment of this application, in step S30, a self-supervised depth estimation network is trained based on the loss function to achieve monocular image depth estimation, including:
[0055] A self-supervised depth estimation network is constructed, which includes the depth network, the pose network, the semantic segmentation model, and the monocular depth estimation model.
[0056] Construct a training dataset, which includes: images, semantic segmentation maps of the images, and prior depth maps;
[0057] The self-supervised depth estimation network is trained based on the training dataset, and the network parameters are optimized based on the loss function.
[0058] Monocular image depth estimation is achieved using the self-supervised depth estimation network.
[0059] A second aspect of this application provides a monocular image depth estimation system, based on the monocular image depth estimation method described above, comprising:
[0060] A deep network construction module is used to construct a deep network and obtain a depth map of an image through the deep network.
[0061] The loss function construction module is used to construct a loss function based on the depth map. The loss function includes a photometric consistency loss function, an edge smoothness loss function, and a confidence depth loss function.
[0062] The network training module is used to train a self-supervised depth estimation network based on the loss function to achieve monocular image depth estimation.
[0063] A third aspect of this application provides a computer-readable medium storing computer-executable instructions for performing the monocular image depth estimation method described above.
[0064] A fourth aspect of this application provides a computing device, comprising:
[0065] At least one processor, and a memory communicatively connected to said at least one processor; wherein,
[0066] The memory stores instructions that can be executed by the at least one processor, the instructions being used to perform the monocular image depth estimation method as described above.
[0067] The invention has at least the following beneficial technical effects:
[0068] The monocular image depth estimation method of this application can solve the failure problem of traditional methods in areas with discontinuous depths at different semantic boundaries, and can enhance the estimation accuracy of low-texture small targets or texture artifact regions in depth estimation. Attached Figure Description
[0069] Figure 1 This is a flowchart of a monocular image depth estimation method according to one embodiment of this application;
[0070] Figure 2 This is a schematic diagram of a deep network according to one embodiment of this application;
[0071] Figure 3 This is a flowchart illustrating the loss function construction process of one implementation of this application;
[0072] Figure 4 This is a schematic diagram of an attitude network according to one embodiment of this application;
[0073] Figure 5 This is a schematic diagram of a monocular image depth estimation system according to one embodiment of this application;
[0074] Figure 6 This is a schematic diagram of the hardware structure of a computing device for implementing a monocular image depth estimation method according to one embodiment of this application. Detailed Implementation
[0075] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below with reference to the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The described embodiments are some, but not all, embodiments of this application. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application. The embodiments of this application will be described in detail below with reference to the accompanying drawings.
[0076] The following is in conjunction with the appendix Figures 1 to 6 This application will be described in further detail.
[0077] The first aspect of this application provides a method for depth estimation in monocular images, such as... Figure 1 As shown, it includes the following steps:
[0078] S10. Construct a deep network to obtain the depth map of the image.
[0079] S20. Construct a loss function based on the depth map. The loss function includes the photometric consistency loss function, the edge smoothness loss function, and the confidence depth loss function.
[0080] S30. A self-supervised depth estimation network is trained based on the loss function to achieve monocular image depth estimation.
[0081] The monocular image depth estimation method of this application firstly, in S10, constructs a deep network, which is used to estimate the depth information of the input image. In this embodiment, it is assumed that the input RGB image is... Where H is the height of the image and W is the width of the image; for example Figure 2 As shown, the deep network adopts a U-Net structure with ResNet-50 as the encoder. The encoder part processes the input image I through a series of convolutions and pooling operations to extract features at different scales; the decoder part, through upsampling and convolution operations, combined with the skip connection information of the encoder, gradually restores the resolution of the original image, and the deep network outputs a depth map.
[0082] In the monocular image depth estimation method of this application, loss function is constructed in step S20. For example... Figure 3 As shown, the construction process includes:
[0083] S21. Construct a pose network to obtain the relative camera pose of two adjacent frames. Construct a photometric consistency loss function based on the depth map and the relative camera pose.
[0084] In a preferred embodiment of this application, the process of constructing the photometric consistency loss function is as follows:
[0085] Construct a pose network to obtain the relative camera pose of two adjacent frames;
[0086] When constructing the photometric consistency loss function, it is necessary to first estimate the I values of two adjacent frames using a pose network. t and I t+1 The relative camera pose between them. In this embodiment, as... Figure 4 As shown, the pose network adopts an improved ResNet-18 structure, changing the number of input channels of the first convolutional kernel of the ResNet-18 structure to 6 to accommodate the concatenated input. After a series of convolutional operations, it outputs 6 degrees of freedom relative to the camera pose through a fully connected layer. This represents the rigid body transformation from frame t to t+1, achieving pose regression.
[0087] A pose network can be represented as:
[0088]
[0089] A synthetic target view image is obtained by combining the depth map output by the deep network and the relative camera pose output by the pose network.
[0090] Based on the differences between the synthesized target view image and the measured target view image, a photometric consistency loss function is constructed:
[0091]
[0092] in, For photometric consistency loss, SSIM is used to calculate structural similarity. To synthesize the target view image, I t The image represents the measured target view, α represents the balancing weight, and ||*|| represents the computational norm.
[0093] S22. Construct a semantic segmentation model. Divide the image into regions using the semantic segmentation model to obtain semantic segmentation regions of different semantic categories. Obtain the set of pixels corresponding to the semantic categories from the semantic segmentation regions. Construct an edge smoothness loss function based on the depth map and the set of pixels.
[0094] In a preferred embodiment of this application, the process of constructing the edge smoothness loss function is as follows:
[0095] The semantic segmentation model (SegNet) is used to divide the image into regions for different objects, resulting in semantic segmentation regions of different semantic categories in the semantic segmentation map. The set of all pixels belonging to semantic category k is obtained, denoted as:
[0096] Ω k ={(i,j)|S ij =k}
[0097] An edge smoothing loss function is constructed based on the depth map gradient to constrain the depth value variations of adjacent pixels and avoid large fluctuations. The edge smoothing loss applies edge-aware smoothing constraints to the normalized inverse depth map.
[0098] Solve the depth map gradient from the depth map;
[0099] Normalize the gradient of the depth map;
[0100]
[0101] Where, d * Here, d represents the normalized depth map gradient. This represents the mean of the depth map gradient.
[0102] Based on the gradient of the normalized depth map, construct the edge smoothness loss function:
[0103]
[0104] in, For edge smoothness loss, For the horizontal gradient operator, Here, E(p) is the vertical gradient operator, E(p) is the intensity value of the image at pixel p, |*| is used to calculate the absolute value, ||*|| is used to calculate the norm, Ω is the set of pixels, and N is the number of pixels.
[0105] The exponential term suppresses the smoothing penalty in high-texture areas, avoiding over-smoothing of object edges.
[0106] The set of pixels Ω for semantic category k k ={(i,j)|S ij For each pixel (i,j) in {k}, calculate the edge smoothness loss based on the depth map gradient and the image gradient weights respectively:
[0107]
[0108] in, For the edge smoothness loss of semantic category k, N k The number of pixels for semantic category k;
[0109] The edge smoothness loss for all semantic categories is summed to obtain the edge smoothness loss function for the entire image based on semantic segmentation:
[0110]
[0111] in, The edge smoothness loss is based on semantic segmentation.
[0112] S23. Obtain the pre-trained monocular depth estimation model, obtain the prior depth map of the image through the monocular depth estimation model, and sample the prior depth map and the depth map to obtain sampling point pairs. Construct the confidence depth loss function based on the sampling point pairs.
[0113] In a preferred embodiment of this application, the confidence depth loss function construction process is as follows:
[0114] A confidence depth loss function is constructed based on sampling point pairs of different semantic categories to constrain the depth ordinal relationship of distant low-texture regions. Random point pairs are sampled for different semantic segmentation regions, and combined with globally random sampling point pairs, the network is forced to learn the correct depth ordinal relationship in different semantic segmentation regions, constraining the depth prediction of distant low-texture regions to be consistent with the depth relationship of nearby high-texture regions.
[0115] Obtain a pre-trained monocular depth estimation model (Depth-e), and use the monocular depth estimation model to obtain the prior depth map of the image;
[0116] Random sampling is performed on the prior depth map region corresponding to the semantic segmentation region of semantic category k in the image to obtain the first sampling point. The predicted depth value of the first sampling point is...
[0117] The same location corresponding to the first sampling point in the depth map is sampled to obtain the second sampling point. The predicted depth value of the second sampling point is...
[0118] The predicted depth value is obtained based on the first and second sampling points. The sampling point pair.
[0119] The confidence depth loss function is constructed based on the sampling point pairs, including:
[0120] The confidence depth loss function is:
[0121]
[0122] in, For confidence depth loss function, The depth order labels are obtained by inducing the prior depth map.
[0123] Determine depth order labels based on prior depth ratios:
[0124]
[0125] in, Let τ be the prior depth of the sampling point pair, and τ be the adjustment coefficient.
[0126] It is understandable that the predicted depth value of the sampling point pair is predicted by the deep network, and the prior depth of the sampling point pair is predicted by the pre-trained monocular depth estimation model.
[0127] Experiments revealed that when the semantic category k is vehicles and ground, the optimal value of the adjustment coefficient τ is between 0.04 and 0.05, and when the semantic category k is bridges and trees, the optimal value of the adjustment coefficient τ is between 0.13 and 0.16.
[0128] The confidence depth loss function corresponding to semantic category k is:
[0129]
[0130] in, Let k be the confidence depth loss function corresponding to semantic category k. The number of sample point pairs for semantic category k;
[0131] By assigning corresponding weight coefficients to different semantic categories k, the confidence depth loss function for semantic constraints is obtained:
[0132]
[0133] Among them, L CDS The confidence depth loss function for semantic constraints. represents the weight coefficient corresponding to semantic category k.
[0134] In one embodiment of this application, the semantic category k includes vehicles, ground, bridges, and trees, and the confidence depth loss function for semantic constraints is:
[0135]
[0136] S24. The loss function is obtained by adding the photometric consistency loss function, the edge smoothness loss function, and the confidence depth loss function.
[0137] Based on the above description, the overall loss function of the deep network is obtained as follows:
[0138]
[0139] In the monocular image depth estimation method of this application, in step S30, a self-supervised depth estimation network is trained based on a loss function to achieve monocular image depth estimation. Specifically, this includes:
[0140] Construct a self-supervised depth estimation network, which includes a deep network, a pose network, a semantic segmentation model, and a monocular depth estimation model;
[0141] Construct a training dataset, which includes: images, semantic segmentation maps of the images, and prior depth maps;
[0142] The self-supervised depth estimation network is trained based on the training dataset, and the network parameters are optimized based on the loss function.
[0143] Monocular image depth estimation is achieved using a self-supervised depth estimation network.
[0144] In a preferred embodiment of this application, when constructing the training dataset, image data of consecutive frames is acquired, the images are downsampled, and then the image scene is semantically segmented using a semantic segmentation model. Multiple semantic categories k-labels are selected, such as semantic segmentation maps for vehicles, road surfaces, bridge surfaces, and trees. A prior depth map is obtained through a pre-trained monocular depth estimation model. During network training, three consecutive image frames and their corresponding semantic segmentation maps and prior depth maps are input as a set, and data augmentation methods are employed during the training process.
[0145] The monocular image depth estimation method proposed in this application designs an edge smoothness loss function based on semantic segmentation, which is calculated independently for different semantic categories, thus solving the failure problem of traditional methods in depth discontinuities at different semantic boundaries. A confidence depth loss function based on semantic constraints for different semantic segmentation regions is established, which has significant advantages in edge accuracy for distant ground objects and accuracy at near and far distances, while avoiding large errors in depth estimation in low-texture regions.
[0146] Based on the above-described monocular image depth estimation method, a second aspect of this application provides a monocular image depth estimation system, such as... Figure 5 As shown, it includes:
[0147] The deep network construction module 10 is used to construct a deep network and obtain the depth map of the image through the deep network.
[0148] The loss function construction module 20 is used to construct a loss function based on the depth map. The loss function includes a photometric consistency loss function, an edge smoothness loss function, and a confidence depth loss function.
[0149] The network training module 30 is used to train a self-supervised depth estimation network based on a loss function to achieve monocular image depth estimation.
[0150] A third aspect of this application provides a computer-readable medium storing computer-executable instructions for performing the monocular image depth estimation method described above.
[0151] For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this invention, the functions of each module (or unit) can be implemented in one or more software or hardware components.
[0152] Having introduced the monocular image depth estimation method, system, and readable medium according to exemplary embodiments of the present invention, a computing device according to another exemplary embodiment of the present invention will now be described.
[0153] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: entirely hardware implementations, entirely software implementations (including firmware, microcode, etc.), or implementations combining hardware and software aspects, collectively referred to herein as “circuits,” “modules,” or “systems.”
[0154] In some possible implementations, the computing device according to the present invention may include at least one processing unit and at least one storage unit. The storage unit stores program code that, when executed by the processing unit, causes the processing unit to perform the steps in the monocular image depth estimation method according to various exemplary embodiments of the present invention described above.
[0155] The following reference Figure 6 To describe the computing device 40 according to this embodiment of the present invention. Figure 6 The computing device 40 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.
[0156] like Figure 6 As shown, the computing device 40 is manifested in the form of a general-purpose computing device. The components of the computing device 40 may include, but are not limited to: at least one processing unit 41, at least one storage unit 42, and a bus 43 connecting different system components (including storage unit 42 and processing unit 41).
[0157] Bus 43 represents one or more of several bus structures, including a memory bus or memory controller, peripheral bus, processor, or a local bus using any of the multiple bus structures.
[0158] Storage unit 42 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 421 and / or cache memory 422, and may further include read-only memory (ROM) 423.
[0159] Storage unit 42 may also include a program / utility 425 having a set (at least one) of program modules 424, such program modules 424 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.
[0160] The computing device 40 can also communicate with one or more external devices 44 (e.g., keyboard, pointing device, etc.), one or more devices that enable a user to interact with the computing device 40, and / or any device that enables the computing device 40 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 45. Furthermore, the computing device 40 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 46. As shown, network adapter 46 communicates with other modules for the computing device 40 via bus 43. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the computing device 40, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0161] In some possible implementations, various aspects of the monocular image depth estimation method provided by the present invention can also be implemented in the form of a program product, which includes program code that, when the program product is run on a computer device, causes the computer device to perform the steps in the monocular image depth estimation method according to various exemplary embodiments of the present invention described above.
[0162] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0163] The program product of the monocular image depth estimation method according to embodiments of the present invention can be a portable compact disc read-only memory (CD-ROM) and include program code, and can run on a computing device. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0164] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take many forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0165] The program code contained on the readable medium can be transmitted using any suitable medium, including—but not limited to—wireless, wired, fiber optic, RF, etc., or any suitable combination thereof. The program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages—such as Java, C++, etc.—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0166] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.
[0167] Furthermore, although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.
[0168] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0169] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0170] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0171] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0172] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method for depth estimation in monocular images, characterized in that, include: S10. Construct a deep network and obtain a depth map of the image through the deep network; S20. Construct a loss function based on the depth map, the loss function including a photometric consistency loss function, an edge smoothness loss function, and a confidence depth loss function; S30. Train a self-supervised depth estimation network based on the loss function to achieve monocular image depth estimation.
2. The monocular image depth estimation method according to claim 1, characterized in that, In S20, a loss function is constructed based on the depth map, including: S21. Construct a pose network to obtain the relative camera pose of two adjacent frames of images, and construct a photometric consistency loss function based on the depth map and the relative camera pose. S22. Construct a semantic segmentation model, divide the image into regions using the semantic segmentation model to obtain semantic segmentation regions of different semantic categories, obtain the set of pixels corresponding to the semantic categories from the semantic segmentation regions, and construct an edge smoothness loss function based on the depth map and the set of pixels. S23. Obtain a pre-trained monocular depth estimation model, obtain a prior depth map of the image through the monocular depth estimation model, and sample the prior depth map and the depth map to obtain sampling point pairs, and construct a confidence depth loss function based on the sampling point pairs. S24. The loss function is obtained by adding the photometric consistency loss function, the edge smoothness loss function and the confidence depth loss function.
3. The monocular image depth estimation method according to claim 2, characterized in that, In S21, a pose network is constructed to obtain the relative camera pose of two adjacent frames. Based on the depth map and the relative camera pose, a photometric consistency loss function is constructed, including: Construct a pose network to obtain the relative camera pose of two adjacent frames; A synthetic target view image is obtained based on the depth map and the relative camera pose; Based on the difference between the synthesized target view image and the measured target view image, a photometric consistency loss function is constructed: in, For photometric consistency loss, SSIM is used to calculate structural similarity. To synthesize the target view image, I t The image represents the measured target view, α represents the balancing weight, and ||*|| represents the computational norm.
4. The monocular image depth estimation method according to claim 3, characterized in that, In S22, an edge smoothness loss function is constructed based on the depth map and the set of pixels, including: The depth map gradient is calculated based on the depth map. The gradient of the depth map is normalized; Where, d * Here, d represents the normalized depth map gradient. This represents the mean of the depth map gradient. Based on the normalized depth map gradient, construct the edge smoothness loss function: in, For edge smoothness loss, For the horizontal gradient operator, Here, E(p) is the vertical gradient operator, E(p) is the intensity value of the image at pixel p, |*| is used to calculate the absolute value, ||*|| is used to calculate the norm, Ω is the set of pixels, and N is the number of pixels. The set of pixels Ω for semantic category k k ={(i,j)|S ij For each pixel (i,j) in {k}, calculate the edge smoothness loss based on the depth map gradient and the image gradient weights respectively: in, For the edge smoothness loss of semantic category k, N k The number of pixels for semantic category k; The edge smoothness loss for all semantic categories is summed to obtain the edge smoothness loss function for the entire image based on semantic segmentation: in, The edge smoothness loss is based on semantic segmentation.
5. The monocular image depth estimation method according to claim 4, characterized in that, In S23, a pre-trained monocular depth estimation model is obtained, a prior depth map of the image is obtained through the monocular depth estimation model, and sampling is performed on the prior depth map and the depth map to obtain sampling point pairs, including: Obtain a pre-trained monocular depth estimation model, and use the monocular depth estimation model to obtain the prior depth map of the image; Random sampling is performed on the prior depth map region corresponding to the semantic segmentation region of semantic category k in the image to obtain a first sampling point, and the predicted depth value of the first sampling point is... The same location corresponding to the first sampling point in the depth map is sampled to obtain a second sampling point, and the predicted depth value of the second sampling point is... The predicted depth value is obtained based on the first sampling point and the second sampling point. The sampling point pair.
6. The monocular image depth estimation method according to claim 5, characterized in that, In S23, the confidence depth loss function is constructed based on the sampling point pairs, including: The confidence depth loss function is: in, Let l be the confidence depth loss function, and l be the depth order label induced by the prior depth map; Determine depth order labels based on prior depth ratios: in, Let τ be the prior depth of the sampling point pair, and τ be the adjustment coefficient. The confidence depth loss function corresponding to semantic category k is: in, Let k be the confidence depth loss function corresponding to semantic category k. The number of sample point pairs for semantic category k; By assigning corresponding weight coefficients to different semantic categories k, the confidence depth loss function for semantic constraints is obtained: Among them, L CDS The confidence depth loss function for semantic constraints. represents the weight coefficient corresponding to semantic category k.
7. The monocular image depth estimation method according to claim 6, characterized in that, In S30, a self-supervised depth estimation network is trained based on the loss function to achieve monocular image depth estimation, including: A self-supervised depth estimation network is constructed, which includes the depth network, the pose network, the semantic segmentation model, and the monocular depth estimation model. Construct a training dataset, which includes: images, semantic segmentation maps of the images, and prior depth maps; The self-supervised depth estimation network is trained based on the training dataset, and the network parameters are optimized based on the loss function. Monocular image depth estimation is achieved using the self-supervised depth estimation network.
8. A monocular image depth estimation system, based on the monocular image depth estimation method according to any one of claims 1 to 7, characterized in that, include: A deep network construction module is used to construct a deep network and obtain a depth map of an image through the deep network. The loss function construction module is used to construct a loss function based on the depth map. The loss function includes a photometric consistency loss function, an edge smoothness loss function, and a confidence depth loss function. The network training module is used to train a self-supervised depth estimation network based on the loss function to achieve monocular image depth estimation.
9. A computer-readable medium storing computer-executable instructions, characterized in that, The computer-executable instructions are used to perform the monocular image depth estimation method as described in any one of claims 1 to 7.
10. A computing device, characterized in that, include: At least one processor, and a memory communicatively connected to said at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being used to perform the monocular image depth estimation method as described in any one of claims 1 to 7.