Multi-model depth map estimation integrated decision method and device and storage medium

Abstract

Embodiments of the application disclose a multi-model depth map estimation comprehensive decision method, device and storage medium, wherein the multi-model depth map estimation comprehensive decision method comprises: acquiring a plurality of two-dimensional images, obtaining different depth maps of each two-dimensional image corresponding to the two-dimensional images through a plurality of different to-be-selected deep learning models; selecting from sparse points visible in the two-dimensional images to obtain a sparse point set, projecting the sparse points in the sparse point set onto the two-dimensional images to obtain a first depth value of each projection point based on the corresponding sparse point; acquiring a second depth value of the projection point corresponding to the depth map, obtaining a loss value of the corresponding depth map on all the two-dimensional images based on the first depth value and the second depth value, and selecting the deep learning model corresponding to the depth map with the minimum loss value as the optimal model to complete multi-model depth map estimation comprehensive decision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a multi-model depth map estimation comprehensive decision method and device and storage medium. BACKGROUND

[0002] The method of depth learning is usually used to train in a supervised manner, which limits the generalization ability of the depth learning model for different scenes. For scenes with large differences from the training data, the depth map estimation ability will decrease significantly. SUMMARY

[0003] The purpose of the embodiments of the present application is to provide a multi-model depth map estimation comprehensive decision method and device and storage medium to solve the problem of insufficient generalization ability of the depth map estimation method based on depth learning in the prior art.

[0004] To achieve the above purpose, the embodiments of the present application provide a multi-model depth map estimation comprehensive decision method, comprising: obtaining a plurality of two-dimensional images, and obtaining different depth maps corresponding to each of the two-dimensional images through a plurality of different depth learning models to be selected;

[0005] Selecting sparse points from the sparse points visible in the two-dimensional images to obtain a sparse point set, projecting the sparse points in the sparse point set onto the two-dimensional images to obtain a first depth value of each projection point based on the corresponding sparse point;

[0006] Obtaining a second depth value of the projection point corresponding to the depth map, and obtaining a loss value of the corresponding depth map on all the two-dimensional images based on the first depth value and the second depth value, and selecting the depth learning model corresponding to the depth map with the smallest loss value as the best model to complete the multi-model depth map estimation comprehensive decision.

[0007] Optionally, the depth learning model comprises MVSNET, P-MVSNET and / or R-MVSNET.

[0008] Optionally, the projecting the sparse points in the sparse point set onto the two-dimensional images comprises:

[0009] Projecting the sparse points in the sparse point set onto the two-dimensional images based on the pose information of the two-dimensional images;

[0010] Wherein, the projection process of the sparse points onto the two-dimensional images comprises converting the sparse points in the sparse point set to normalized plane coordinates through an extrinsic parameter matrix, and then converting to pixel coordinates of the two-dimensional images through an intrinsic parameter matrix.

[0011] Optionally, obtaining the loss value of the corresponding depth map across all the two-dimensional images based on the first depth value and the second depth value includes:

[0012] Through formula The loss value of the o-th depth map for the two-dimensional image is obtained, where depth i This represents the first depth value of the i-th projection point based on the corresponding sparse point. The second depth value of the depth map corresponding to the i-th projection point is PN, which represents the number of sparse points in the sparse point set.

[0013] Through formula The loss values ​​of the corresponding depth map are obtained on all the two-dimensional images, where N represents the number of the acquired two-dimensional images.

[0014] To achieve the above objectives, this application also provides a multi-model depth map estimation and comprehensive decision-making apparatus, comprising: a memory; and

[0015] A processor connected to the memory, the processor being configured to:

[0016] Acquire several two-dimensional images, and obtain different depth maps for each of the two-dimensional images through several different deep learning models to be selected;

[0017] A sparse point set is obtained by selecting sparse points visible in the two-dimensional image, and the sparse points in the sparse point set are projected onto the two-dimensional image to obtain a first depth value for each projected point based on the corresponding sparse point.

[0018] The second depth value of the projection point corresponding to the depth map is obtained. Based on the first depth value and the second depth value, the loss value of the corresponding depth map on all the two-dimensional images is obtained. The deep learning model corresponding to the depth map with the smallest loss value is selected as the best model to complete the comprehensive decision of multi-model depth map estimation.

[0019] To achieve the above objectives, this application also provides a computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a machine, implements the steps of the method described above.

[0020] The embodiments of this application have the following advantages:

[0021] This application provides a multi-model depth map estimation comprehensive decision-making method, including: acquiring several two-dimensional images; obtaining different depth maps for each of the two-dimensional images through several different deep learning models to be selected; selecting sparse points visible in the two-dimensional images to obtain a sparse point set; projecting the sparse points in the sparse point set onto the two-dimensional images to obtain a first depth value for each projected point based on the corresponding sparse point; obtaining a second depth value of the projection point corresponding to the depth map; obtaining a loss value of the corresponding depth map on all the two-dimensional images based on the first and second depth values; and selecting the deep learning model corresponding to the depth map with the smallest loss value as the optimal model to complete the multi-model depth map estimation comprehensive decision-making.

[0022] Using the above method, for multiple models performing depth map estimation, a loss value is obtained based on the second depth value and the first depth value of each projection point on the two-dimensional image based on the corresponding sparse point. The loss values ​​are compared, and the deep learning model corresponding to the smallest loss value is selected as the best model for the current scene. This model has the best generalization ability, thus solving the problem of insufficient generalization ability of deep learning-based depth map estimation methods. Attached Figure Description

[0023] To more clearly illustrate the embodiments of this application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0024] Figure 1 A flowchart of a multi-model depth map estimation integrated decision-making method provided in this application embodiment;

[0025] Figure 2 A schematic diagram of sparse point projection onto a two-dimensional image for a multi-model depth map estimation integrated decision-making method provided in an embodiment of this application;

[0026] Figure 3 This is a block diagram of a multi-model depth map estimation integrated decision-making device provided in an embodiment of this application. Detailed Implementation

[0027] The following specific embodiments illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] Furthermore, the technical features involved in the different embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0029] One embodiment of this application provides a multi-model depth map estimation integrated decision-making method, referencing... Figure 1 , Figure 1 The flowchart provided in one embodiment of this application illustrates a multi-model depth map estimation integrated decision method. It should be understood that the method may also include additional boxes not shown and / or the boxes shown may be omitted, and the scope of this application is not limited in this respect.

[0030] In step 101, several two-dimensional images are acquired, and different depth maps for each of the two-dimensional images are obtained through several different deep learning models to be selected.

[0031] Specifically, for scenarios with significant deviations, deep learning models are trained to form multi-model structures with different training data, such as MVSNET, P-MVSNET, and R-MVSNET. Initially, N two-dimensional images are randomly selected, where N ≤ NT (in some embodiments, the threshold range of NT is [1, 10], in units of images), and the image set is I = {I1, I2, I3, ...}. Each two-dimensional image is processed by a different selected deep learning model, such as... Figure 2 As shown, for the m-th two-dimensional image I m Different depth maps are obtained through different deep learning models.

[0032] In step 102, a sparse point set is obtained by selecting from the sparse points visible in the two-dimensional image, and the sparse points in the sparse point set are projected onto the two-dimensional image to obtain a first depth value for each projected point based on the corresponding sparse point.

[0033] Specifically, refer to Figure 2 From two-dimensional image I mAmong the sparse points visible in the image (each pixel in the image corresponds to only a portion of the sparse points, and the sparse points are the coordinate points in the three-dimensional sparse point cloud; the sparse points visible in the two-dimensional image are the sparse points corresponding to the pixels in the two-dimensional image), PN sets of sparse points {P1, P2, P3, ...} are randomly selected, where PN ≤ PNT (in some embodiments, the threshold range of PNT is [50, 500]).

[0034] In some embodiments, projecting sparse points from the sparse point set onto the two-dimensional image includes:

[0035] Based on the pose information of the two-dimensional image, the sparse points in the sparse point set are projected onto the two-dimensional image;

[0036] The projection process of sparse points onto the two-dimensional image includes transforming the sparse points in the sparse point set to normalized plane coordinates through an extrinsic parameter matrix, and then transforming them to pixel coordinates of the two-dimensional image through an intrinsic parameter matrix.

[0037] Specifically, based on the pose information of the two-dimensional image, it is projected onto the two-dimensional image ( Figure 2 In the image scale (h, w), in some embodiments, the projection process involves transforming sparse points to normalized planar coordinates through an extrinsic parameter matrix, and then transforming them to pixel coordinates in a two-dimensional image through an intrinsic parameter matrix.

[0038] refer to Figure 2 The projection points are p1, p2, p3, ..., and the first depth value of each projection point based on the sparse point is depth1, depth2, depth3, ... (a sparse point is a three-dimensional point, which also has a depth value called the "depth value of the sparse point"). i It is the depth value obtained based on sparse points, i.e., the first depth value.

[0039] In step 103, the second depth value of the projection point corresponding to the depth map is obtained. Based on the first depth value and the second depth value, the loss value of the corresponding depth map on all the two-dimensional images is obtained. The deep learning model corresponding to the depth map with the smallest loss value is selected as the best model to complete the multi-model depth map estimation comprehensive decision.

[0040] Specifically, the projection point corresponds to the second depth value of the o-th depth map. (The second depth value is obtained by projecting coordinates onto the depth map). The depth map generated by the 0th deep learning model is the 0th depth map, based on sparse points in I... mThe two-dimensional projection coordinates on the two-dimensional image can be used to find the corresponding second depth value in the o-th depth map. This is different from the first depth value of sparse points, which is a depth value obtained based on the three-dimensional coordinates of the sparse points.

[0041] In some embodiments, obtaining the loss value of the corresponding depth map across all the two-dimensional images based on the first depth value and the second depth value includes:

[0042] Through formula The loss value of the o-th depth map for the two-dimensional image is obtained, where depth i This represents the first depth value of the i-th projection point based on the corresponding sparse point. The second depth value of the depth map corresponding to the i-th projection point is PN, which represents the number of sparse points in the sparse point set.

[0043] Through formula The loss values ​​of the corresponding depth map are obtained on all the two-dimensional images, where N represents the number of the acquired two-dimensional images.

[0044] Specifically, through That is, to obtain the two-dimensional image I m The loss value of the o-th depth map; through Obtain the loss values ​​of the corresponding depth map on all randomly selected 2D images, and calculate the minimum cost. o The corresponding deep learning model is selected as the best model for the current scene, and its estimated depth map is used.

[0045] Using the above method, for multiple models performing depth map estimation, a loss value is obtained based on the second depth value and the first depth value of each projection point on the two-dimensional image based on the corresponding sparse point. The loss values ​​are compared, and the deep learning model corresponding to the smallest loss value is selected as the best model for the current scene. This model has the best generalization ability, thus solving the problem of insufficient generalization ability of deep learning-based depth map estimation methods.

[0046] Figure 3 A block diagram of a multi-model depth map estimation and comprehensive decision-making device provided in this application embodiment. The device includes:

[0047] The memory 201; and the processor 202 connected to the memory 201, the processor 202 being configured to: acquire a plurality of two-dimensional images, and obtain different depth maps for each of the two-dimensional images through a plurality of different deep learning models to be selected;

[0048] A sparse point set is obtained by selecting sparse points visible in the two-dimensional image, and the sparse points in the sparse point set are projected onto the two-dimensional image to obtain a first depth value for each projected point based on the corresponding sparse point.

[0049] The second depth value of the projection point corresponding to the depth map is obtained. Based on the first depth value and the second depth value, the loss value of the corresponding depth map on all the two-dimensional images is obtained. The deep learning model corresponding to the depth map with the smallest loss value is selected as the best model to complete the comprehensive decision of multi-model depth map estimation.

[0050] In some embodiments, the processor 202 is further configured such that the deep learning model includes MVSNET, P-MVSNET, and / or R-MVSNET.

[0051] In some embodiments, the processor 202 is further configured to: project sparse points from the sparse point set onto the two-dimensional image, including:

[0052] Based on the pose information of the two-dimensional image, the sparse points in the sparse point set are projected onto the two-dimensional image;

[0053] The projection process of sparse points onto the two-dimensional image includes transforming the sparse points in the sparse point set to normalized plane coordinates through an extrinsic parameter matrix, and then transforming them to pixel coordinates of the two-dimensional image through an intrinsic parameter matrix.

[0054] In some embodiments, the processor 202 is further configured to: obtain the loss value of the corresponding depth map on all the two-dimensional images based on the first depth value and the second depth value, including:

[0055] Through formula The loss value of the o-th depth map for the two-dimensional image is obtained, where depth i This represents the first depth value of the i-th projection point based on the corresponding sparse point. The second depth value of the depth map corresponding to the i-th projection point is PN, which represents the number of sparse points in the sparse point set.

[0056] Through formula The loss values ​​of the corresponding depth map are obtained on all the two-dimensional images, where N represents the number of the acquired two-dimensional images.

[0057] For specific implementation methods, please refer to the aforementioned method embodiments, which will not be repeated here.

[0058] This application may be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this application.

[0059] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0060] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0061] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from the computer-readable program instructions. These electronic circuits can execute the computer-readable program instructions to implement various aspects of this application.

[0062] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0063] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0064] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0065] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0066] Note that, unless otherwise explicitly stated, all features disclosed in this specification (including any appended claims, abstract, and drawings) may be replaced by alternative features for achieving the same, equivalent, or similar purpose. Therefore, unless explicitly stated otherwise, each disclosed feature is merely one example of a set of equivalent or similar features. Where used, "further," "preferably," "even further," and "more preferably" are simple starting points for describing another embodiment based on the foregoing embodiments, the combination of which with the foregoing embodiments constitutes the complete configuration of another embodiment. Any combination of several "further," "preferably," "even further," or "more preferably" settings following the same embodiment constitutes yet another embodiment.

[0067] Although this application has been described in detail above with general descriptions and specific embodiments, some modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of this application fall within the scope of protection claimed in this application.

Claims

1. A multi-model depth map estimation consensus decision method, characterized in that, The method comprises: obtaining a plurality of two-dimensional images, and obtaining different depth maps corresponding to each of the two-dimensional images through a plurality of different to-be-selected deep learning models; selecting sparse points from sparse points visible in the two-dimensional images to obtain a sparse point set, projecting the sparse points in the sparse point set onto the two-dimensional images to obtain a first depth value of each projection point based on a corresponding sparse point; obtaining a second depth value corresponding to the projection point to the depth map, obtaining a loss value of the corresponding depth map on all the two-dimensional images based on the first depth value and the second depth value, and selecting the deep learning model corresponding to the depth map with the minimum loss value as the optimal model to complete the multi-model depth map estimation comprehensive decision.

2. The multi-model depth map estimation comprehensive decision method according to claim 1, wherein the deep learning model comprises MVSNET, P-MVSNET and / or R-MVSNET. The projection of the sparse points in the sparse point set onto the two-dimensional images comprises:

3. The multi-model depth map estimation consensus decision method of claim 1, wherein, projecting the sparse points in the sparse point set onto the two-dimensional images based on pose information of the two-dimensional images; wherein the projection of the sparse points onto the two-dimensional images comprises converting the sparse points in the sparse point set to normalized plane coordinates through an extrinsic parameter matrix, and then converting the normalized plane coordinates to pixel coordinates of the two-dimensional images through an intrinsic parameter matrix. The processor is further configured to:

4. The multi-model depth map estimation consensus decision method of claim 1, wherein, obtain a plurality of two-dimensional images, and obtain different depth maps corresponding to each of the two-dimensional images through a plurality of different to-be-selected deep learning models; The loss value for the oth depth map of the two-dimensional image is obtained by the formula i represents the first depth value of the i th projection point based on the corresponding sparse point, represents the second depth value of the i th projection point corresponding to the o th depth map, and PN represents the number of sparse points in the sparse point set.​ The loss value corresponding to the depth map on all the two-dimensional images is obtained by the formula where N represents the number of the obtained two-dimensional images.

5. A multi-model depth map estimation synthesis decision apparatus characterized by comprising: select sparse points from sparse points visible in the two-dimensional images to obtain a sparse point set, project the sparse points in the sparse point set onto the two-dimensional images to obtain a first depth value of each projection point based on a corresponding sparse point; obtain a second depth value corresponding to the projection point to the depth map, obtain a loss value of the corresponding depth map on all the two-dimensional images based on the first depth value and the second depth value, and select the deep learning model corresponding to the depth map with the minimum loss value as the optimal model to complete the multi-model depth map estimation comprehensive decision. The processor is further configured to: the deep learning model comprises MVSNET, P-MVSNET and / or R-MVSNET. The processor is further configured to: the projection of the sparse points in the sparse point set onto the two-dimensional images comprises: projecting the sparse points in the sparse point set onto the two-dimensional images based on pose information of the two-dimensional images; 6. The multi-model depth map estimation consensus decision apparatus of claim 5, wherein, wherein the projection of the sparse points onto the two-dimensional images comprises converting the sparse points in the sparse point set to normalized plane coordinates through an extrinsic parameter matrix, and then converting the normalized plane coordinates to pixel coordinates of the two-dimensional images through an intrinsic parameter matrix. The processor is further configured to:

7. The multi-model depth map estimation consensus decision apparatus of claim 5, wherein, ​ ​ ​ ​ 8. The multi-model depth map estimation consensus decision apparatus of claim 5, wherein, ​ The loss value of the corresponding depth map on all the two-dimensional images is obtained based on the first depth value and the second depth value, and the loss value comprises: A loss value for the o-th depth map of the two-dimensional image is obtained by the formula wherein depth i denotes the first depth value of the i-th projection point based on the corresponding sparse point, denotes the second depth value of the i-th projection point corresponding to the o-th depth map, and PN denotes the number of sparse points in the set of sparse points. The loss value corresponding to the depth map on all the two-dimensional images is obtained by the formula where N represents the number of the obtained two-dimensional images.

9. A computer storage medium having stored thereon a computer program, characterized in that The computer program is executed by a machine to implement the steps of the method in any one of claims 1 to 4.

Images