Depth estimation scheme evaluation method and device, equipment and storage medium
By evaluating the depth value of the depth map sequence to be evaluated from the visual feature level, the problem of blurred depth map evaluation results and low accuracy in the prior art is solved, and a more accurate evaluation of the depth estimation scheme is achieved.
Patent Information
- Application Number
- CN202311120825.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-31
- Publication Date
- 2025-02-28
AI Technical Summary
The existing depth estimation evaluation technology mainly focuses on the overall accuracy of the depth map, resulting in blurred evaluation results and low accuracy.
By obtaining the depth map sequence to be evaluated, based on the true depth map sequence and the marked multi-view color image sequence, depth value evaluation is performed from the visual feature level, and the depth value evaluation results are obtained, and the evaluation results of the depth estimation scheme are determined.
The accuracy of the assessment of the depth estimation scheme is improved, and a more detailed and accurate assessment of the depth estimation accuracy is provided.
Smart Images
Figure CN119540317A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a method, device, equipment and storage medium for evaluating a depth estimation solution. Background Art
[0002] Depth estimation is a computer vision task that aims to estimate depth from a two-dimensional image. This task requires a color image as input and outputs a depth image. In order to determine whether the depth value estimation of each pixel in the depth image is accurate, the depth value of the depth image can be evaluated.
[0003] In related technologies, the depth estimation evaluation technology used mainly focuses on the overall accuracy of the depth map. Typical indicators include scale invariant error, absolute relative error, root mean squared error, etc., which leads to vague depth map evaluation results and low accuracy. Summary of the invention
[0004] The embodiment of the present application provides a depth estimation solution evaluation method, device, equipment and storage medium, and the technical solution is as follows:
[0005] On the one hand, an embodiment of the present application provides a method for evaluating a depth estimation solution, the method comprising:
[0006] Acquire a depth map sequence to be evaluated, wherein the depth map sequence to be evaluated includes a plurality of depth images to be evaluated at different viewing angles, the depth images to be evaluated correspond one-to-one to color images in a multi-view color image sequence, the color images are obtained by photographing a target scene at different viewing angles, and the depth images to be evaluated are obtained by performing depth estimation on the color images based on a depth estimation scheme to be evaluated;
[0007] Based on the true value depth map sequence and the annotated multi-view color image sequence, a depth value evaluation is performed on the depth map sequence to be evaluated from the visual feature level to obtain a depth value evaluation result of the depth map sequence to be evaluated, wherein the true value depth images in the true value depth map sequence correspond one-to-one to the depth images to be evaluated in the depth map sequence to be evaluated;
[0008] Based on the depth value evaluation results corresponding to the depth map sequence to be evaluated, a scheme evaluation result of the depth estimation scheme to be evaluated is determined, and the scheme evaluation result represents the depth estimation accuracy of the depth estimation scheme to be evaluated.
[0009] On the other hand, an embodiment of the present application provides a depth estimation solution evaluation device, the device comprising:
[0010] A first acquisition module is used to acquire a depth map sequence to be evaluated, wherein the depth map sequence to be evaluated includes a plurality of depth images to be evaluated at different viewing angles, wherein the depth images to be evaluated correspond one-to-one to color images in a multi-view color image sequence, wherein the color images are obtained by photographing a target scene at different viewing angles, and wherein the depth images to be evaluated are obtained by performing depth estimation on the color images based on a depth estimation scheme to be evaluated;
[0011] An evaluation module is used to perform depth value evaluation on the depth map sequence to be evaluated from the visual feature level based on the true value depth map sequence and the annotated multi-view color image sequence, and obtain the depth value evaluation result of the depth map sequence to be evaluated, wherein the true value depth image in the true value depth map sequence corresponds one-to-one to the depth image to be evaluated in the depth map sequence to be evaluated;
[0012] The result determination module is used to determine the scheme evaluation result of the depth estimation scheme to be evaluated based on the depth value evaluation result corresponding to the depth map sequence to be evaluated, and the scheme evaluation result represents the depth estimation accuracy of the depth estimation scheme to be evaluated.
[0013] On the other hand, an embodiment of the present application provides a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the evaluation method of the depth estimation scheme as described in the above aspects.
[0014] On the other hand, an embodiment of the present application provides a computer-readable storage medium, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement the evaluation method of the depth estimation scheme as described in the above aspects.
[0015] On the other hand, an embodiment of the present application provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the evaluation method of the depth estimation solution provided in the above aspect.
[0016] In an embodiment of the present application, in order to evaluate the depth estimation accuracy of the depth estimation scheme to be evaluated, first, a depth map sequence to be evaluated is obtained, and the depth map sequence to be evaluated is obtained by performing depth estimation on each color image in a multi-view color image sequence based on the depth estimation scheme to be evaluated, wherein the color image is obtained by photographing the target scene at different viewing angles, thereby performing depth value evaluation on the depth map sequence to be evaluated from the visual feature level according to the true value depth map sequence and the labeled multi-view color image sequence, that is, obtaining the depth value evaluation result of the depth map sequence to be evaluated, and then determining the scheme evaluation result of the depth estimation scheme to be evaluated according to the depth value evaluation result corresponding to the depth map sequence to be evaluated; adopting the scheme provided in the embodiment of the present application, the depth value evaluation on the depth map sequence to be evaluated is performed from the visual feature level through the true value depth image sequence and the labeled multi-view color image sequence, thereby improving the evaluation accuracy of the depth estimation scheme. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0018] Figure 1 A schematic diagram showing an implementation environment provided by an exemplary embodiment of the present application is shown;
[0019] Figure 2 A flow chart of a method for evaluating a depth estimation solution provided by an exemplary embodiment of the present application is shown;
[0020] Figure 3 A flow chart of a method for evaluating a depth estimation solution provided by another exemplary embodiment of the present application is shown;
[0021] Figure 4 A flowchart of determining a true depth map sequence provided by an exemplary embodiment of the present application is shown;
[0022] Figure 5 A structural block diagram of an evaluation device for a depth estimation solution provided by an exemplary embodiment of the present application is shown;
[0023] Figure 6 A structural block diagram of a computer device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION
[0024] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.
[0025] Please refer to Figure 1 , which shows a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application. The implementation environment includes a terminal 120 and a server 140. The terminal 120 and the server 140 perform data communication via a communication network. Optionally, the communication network can be a wired network or a wireless network, and the communication network can be at least one of a local area network, a metropolitan area network, and a wide area network.
[0026] The terminal 120 is an electronic device installed with an application having a depth estimation scheme evaluation function. The depth estimation scheme evaluation function may be a function of a native application in the terminal, or a function of a third-party application; the electronic device may be a smart phone, a tablet computer, a personal computer, a wearable device, or a vehicle-mounted terminal, etc. Figure 1 In the description, the terminal 120 is taken as a personal computer as an example, but this is not a limitation.
[0027] The server 140 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. In the embodiment of the present application, the server 140 may be a background server of an application having a depth estimation solution evaluation function.
[0028] In one possible implementation, Figure 1 As shown, there is data interaction between the server 140 and the terminal 120. The terminal 120 obtains the depth map sequence to be evaluated, and sends the depth map sequence to be evaluated to the server 140. The server 140 performs depth value evaluation on the depth map sequence to be evaluated from the visual feature level based on the true value depth map sequence and the marked multi-view color image sequence, obtains the depth value evaluation result corresponding to the depth map sequence to be evaluated, and then obtains the scheme evaluation result of the depth estimation scheme to be evaluated, and sends the scheme evaluation result to the terminal 120.
[0029] Please refer to Figure 2 , which shows a flowchart of a depth estimation solution evaluation method provided by an exemplary embodiment of the present application. This embodiment is described by taking the method used in a computer device (including a terminal 120 and / or a server 140) as an example. The method includes the following steps:
[0030] Step 201, obtain a depth map sequence to be evaluated, the depth map sequence to be evaluated includes multiple depth images to be evaluated under different viewing angles, the depth images to be evaluated correspond one-to-one to the color images in the multi-view color image sequence, the color images are obtained by photographing the target scene under different viewing angles, and the depth images to be evaluated are obtained by performing depth estimation on the color images based on the depth estimation scheme to be evaluated.
[0031] Optionally, in order to evaluate the depth estimation accuracy of the depth estimation scheme to be evaluated, the computer device may evaluate the depth value of the depth image to be evaluated obtained by performing depth estimation on the color image based on the depth estimation scheme to be evaluated.
[0032] In one possible implementation, a computer device first determines a set of multi-view color image sequences, wherein the multi-view color image sequence includes multiple color images, and each color image is obtained by photographing a target scene at a different viewing angle. Optionally, the multi-view color image sequence may include multiple color images corresponding to a single scene, and may also include multiple color images corresponding to multiple scenes, respectively. This embodiment of the present application is not limited to this.
[0033] Furthermore, the computer device can perform depth estimation on each color image in the group of multi-view color image sequences through the depth estimation scheme to be evaluated, and obtain a depth image to be evaluated corresponding to each color image, that is, a depth map sequence to be evaluated.
[0034] Step 202, based on the true value depth map sequence and the annotated multi-view color image sequence, perform depth value evaluation on the depth map sequence to be evaluated from the visual feature level to obtain the depth value evaluation result of the depth map sequence to be evaluated, and the true value depth image in the true value depth map sequence corresponds one-to-one to the depth image to be evaluated in the depth map sequence to be evaluated.
[0035] Different from the related art, in which the depth estimation is evaluated for the overall accuracy of the depth image only based on indicators such as scale-invariant error, absolute relative error, root mean square difference, etc., which leads to ignoring the visually significant features in the depth image, thereby affecting the evaluation accuracy of the depth estimation scheme, in the embodiment of the present application, a method of evaluating the depth value of the depth map sequence to be evaluated from the visual feature level is adopted to determine the depth value evaluation result of the depth map sequence to be evaluated.
[0036] In a possible implementation, in order to accurately evaluate depth values based on visual features, a computer device may first obtain a labeled multi-view color image sequence, wherein the labeled multi-view color image sequence refers to a multi-view color image sequence obtained by feature labeling each color image based on visual features.
[0037] Optionally, labeling of each color image according to visual features may be done manually, or by training a feature labeling model and then completing the labeling using the feature labeling model, which is not limited in the embodiments of the present application.
[0038] Optionally, the feature annotation method may be to add a label instance to each color image, and the specific feature annotation method is not limited in the embodiment of the present application.
[0039] In a possible implementation, in order to determine the accuracy of the depth estimation value of each pixel in each depth image to be evaluated, the computer device also needs to obtain a true depth map sequence, wherein each true depth image in the true depth map sequence corresponds one-to-one to the depth image to be evaluated.
[0040] Optionally, the true depth map sequence can be obtained based on a structured light sensor, can be obtained through a lidar, or can be obtained by determining a dense depth map, which is not limited in the embodiments of the present application.
[0041] In a possible implementation, after obtaining a true depth map sequence and a labeled multi-view color image sequence, a computer device can perform depth value evaluation on the depth map sequence to be evaluated from a visual feature level to obtain a depth value evaluation result of the depth map sequence to be evaluated.
[0042] Optionally, the depth value evaluation result may be expressed as a scoring result of the overall depth estimation accuracy of the depth map sequence to be evaluated, or may be other result expression methods, which is not limited in the embodiments of the present application.
[0043] Step 203 : determining a scheme evaluation result of the depth estimation scheme to be evaluated based on the depth value evaluation result corresponding to the depth map sequence to be evaluated, wherein the scheme evaluation result represents the depth estimation accuracy of the depth estimation scheme to be evaluated.
[0044] In a possible implementation, after obtaining the depth value evaluation results corresponding to the depth map sequence to be evaluated, the computer device needs to further determine the scheme evaluation results of the depth estimation scheme to be evaluated, and the scheme evaluation results represent the depth estimation accuracy of the depth estimation scheme to be evaluated.
[0045] Optionally, the scheme evaluation result of the depth estimation scheme to be evaluated is positively correlated with the depth value evaluation result corresponding to the depth map sequence to be evaluated. When the depth value evaluation result corresponding to the depth map sequence to be evaluated indicates that the accuracy of the depth estimation value of the depth map sequence to be evaluated is high, the scheme evaluation result of the depth estimation scheme to be evaluated indicates that the depth estimation accuracy of the depth estimation scheme to be evaluated is high.
[0046] To summarize, in an embodiment of the present application, in order to evaluate the depth estimation accuracy of the depth estimation scheme to be evaluated, first, a depth map sequence to be evaluated is obtained, and the depth map sequence to be evaluated is obtained by performing depth estimation on each color image in a multi-view color image sequence based on the depth estimation scheme to be evaluated, wherein the color image is obtained by photographing the target scene at different view angles, thereby performing depth value evaluation on the depth map sequence to be evaluated from the visual feature level according to the true value depth map sequence and the labeled multi-view color image sequence, that is, obtaining the depth value evaluation result of the depth map sequence to be evaluated, and then determining the scheme evaluation result of the depth estimation scheme to be evaluated according to the depth value evaluation result corresponding to the depth map sequence to be evaluated; adopting the scheme provided in the embodiment of the present application, the depth value evaluation of the depth map sequence to be evaluated is performed from the visual feature level through the true value depth image sequence and the labeled multi-view color image sequence, thereby improving the evaluation accuracy of the depth estimation scheme.
[0047] In a possible implementation, in order to perform depth value evaluation on a depth map sequence to be evaluated from a visual feature level, a computer device may perform depth value evaluation on each depth image to be evaluated in the depth map sequence to be evaluated in turn, thereby obtaining a depth value evaluation result of the entire depth map sequence to be evaluated based on the depth value evaluation results of each depth image to be evaluated.
[0048] Please refer to Figure 3 , which shows a flow chart of a depth estimation solution evaluation method provided by an exemplary embodiment of the present application. This embodiment is described by taking the method used in a computer device as an example. The method includes the following steps:
[0049] Step 301, obtain a depth map sequence to be evaluated, the depth map sequence to be evaluated includes multiple depth images to be evaluated under different viewing angles, the depth images to be evaluated correspond one-to-one to the color images in the multi-view color image sequence, the color images are obtained by photographing the target scene under different viewing angles, and the depth images to be evaluated are obtained by performing depth estimation on the color images based on the depth estimation scheme to be evaluated.
[0050] The specific implementation of step 301 may refer to step 201 , and will not be described in detail in this embodiment.
[0051] Step 302, based on the true value depth map sequence and the labeled multi-view color image sequence, perform depth value evaluation on each depth image to be evaluated in the depth map sequence from the visual feature level, and determine the image-level depth value evaluation result corresponding to each depth image to be evaluated. The image-level depth value evaluation result represents the depth estimation of the depth image to be evaluated at the visual feature level, and the visual features include at least one of edge quality, regional smoothness and temporal stability.
[0052] In one possible implementation, in order to obtain the depth value evaluation results corresponding to the entire depth map sequence to be evaluated, the computer device can first perform depth value evaluation on each depth image to be evaluated in the depth map sequence to be evaluated from the visual feature level, and obtain the image-level depth value evaluation results corresponding to each depth image to be evaluated.
[0053] Optionally, the image-level depth value evaluation result represents the depth estimation of each depth image to be evaluated at the visual feature level, and specifically may be the accuracy of the depth estimation value of each pixel point on the depth image to be evaluated.
[0054] Optionally, in order to more accurately evaluate the depth value of each depth image to be evaluated from the visual feature level, the visual features can also be specifically divided into edge quality, regional smoothness and temporal stability, among which edge quality is used to determine the accuracy of the depth estimation value of each pixel located at the contour; regional smoothness is used to determine the degree of difference in depth estimation between adjacent pixels; and temporal stability is used to determine the changes in the depth estimation values of key pixels in the image between previous and next frames.
[0055] Optionally, the visual feature is edge quality, and the edge quality may include foreground character contour quality and foreground object contour quality. In one possible implementation, the computer device first obtains a multi-view color image sequence annotated with character features and a multi-view color image sequence annotated with object features, and then performs depth value evaluation on each depth image to be evaluated based on the true value depth map sequence and the multi-view color image sequence annotated with character features, and determines a first image-level depth value evaluation result corresponding to the foreground character contour quality of each depth image to be evaluated, and performs depth value evaluation on each depth image to be evaluated based on the true value depth map sequence and the multi-view color image sequence annotated with object features, and determines a second image-level depth value evaluation result corresponding to the foreground object contour quality of each depth image to be evaluated.
[0056] Optionally, the first image-level depth value evaluation results may include the accuracy and recall of the overall contour of the foreground character in the image, the accuracy and recall of the contour of the hollow part of the foreground character, the accuracy and recall of the contour of the foreground character's head, the accuracy and recall of the contour of the foreground character's hand, etc., which is not limited to the embodiments of the present application.
[0057] Optionally, the second image-level depth value evaluation results may include the accuracy and recall of the overall contour of the foreground object in the image, the accuracy and recall of the contour of the hollow part of the foreground object, the accuracy and recall of the contour of the translucent object, the accuracy and recall of the contour of the handheld object, etc., which is not limited to the embodiments of the present application.
[0058] Optionally, the visual feature is regional smoothness. In a possible implementation, the computer device first obtains a multi-view color image sequence annotated with regional features, and then performs depth value evaluation on each depth image to be evaluated based on the true value depth map sequence and the multi-view color image sequence annotated with regional features, and determines the third image-level depth value evaluation result corresponding to each depth image to be evaluated in terms of regional smoothness.
[0059] Optionally, the third image-level depth value evaluation result may include the proportion of erroneous texture produced in the highlight area, background smoothness (the degree to which the background produces abnormal bright spots), relative depth (measuring the order of distance of key features), similar depth (measuring the change in depth value within a similar depth area), depth gradient (measuring the smoothness of depth value in a linear change area), etc., which is not limited to the embodiments of the present application.
[0060] Optionally, the visual feature is temporal stability. In a possible implementation, the computer device first obtains a multi-view color image sequence annotated with temporal features, and then performs depth value evaluation on each depth image to be evaluated based on the true value depth map sequence and the multi-view color image sequence annotated with temporal features, and determines the fourth image-level depth value evaluation result corresponding to each depth image to be evaluated in terms of temporal stability.
[0061] Optionally, the fourth image-level depth value evaluation results may include foreground flickering (the change in the depth of the foreground between previous and next frames in a static foreground scene), background flickering (the change in the depth of the background between previous and next frames in a static background scene), background disturbance (global change in the depth of the background in a static background scene), consistency of key points before and after (consistency of key points before and after frames in a static scene), etc., which are not limited in the embodiments of the present application.
[0062] In a possible implementation, in order to reduce the amount of data, the person features, object features, region features and time series features can be uniformly annotated on a set of multi-view color image sequences by labeling, and different label categories can be distinguished by different label names. Schematically, the multi-instance label format can be expressed as <label>_#,in, <label>Indicates a label, such as human, object, bottle, etc. # is used to distinguish different label instances. For example, if there are two characters in a scene, the labels human_1 and human_2 can be used. For other possible label representation methods, please refer to Table 1, which is not specifically limited in the present embodiment of the application.
[0063] Table 1
[0064]
[0065]
[0066] In a possible implementation, in order to obtain a more accurate true value depth map sequence, the computer device may first obtain a multi-view color image sequence and a video image sequence corresponding to the target scene, wherein the video image sequence is obtained by shooting the target scene left and right along a large baseline, and the target scene is kept in the center of the image during the shooting process. For example, the camera may be controlled to slowly translate left and right along a straight line at a speed of about 20 cm / s in front of the target scene, with a left and right travel of 40 cm, and the target scene may be kept in the center of the image by slightly rotating the camera during the translation process.
[0067] Furthermore, the computer device can determine the camera pose corresponding to each color image in the multi-view color image sequence based on the multi-view color image sequence and the video image sequence. In a possible implementation, the computer device can use the Structure From Motion (SFM) technology to calculate the camera pose corresponding to each image in the multi-view color image sequence and the video image sequence.
[0068] Furthermore, the computer device can obtain a true depth map sequence corresponding to the multi-view color image sequence by multi-view depth estimation according to the multi-view color image sequence and the camera pose corresponding to each color image. In a possible implementation, the computer device can use multi-view stereo vision technology (Multiple View Stereo, MVS) to calculate the dense depth map corresponding to each color image in the multi-view color image sequence, thereby obtaining a true depth map sequence.
[0069] Indicatively, Figure 4 As shown, the computer device first obtains a multi-view color image sequence and a video image sequence corresponding to the target scene, and then uses the SFM technology to perform three-dimensional reconstruction of the feature points of the target scene according to each multi-view image to obtain three-dimensional coordinates, and estimates the camera pose corresponding to each color image, and then generates a dense depth map corresponding to each color image based on the MVS technology.
[0070] Step 303: Determine the depth value evaluation result of the depth image sequence to be evaluated based on the image-level depth value evaluation result corresponding to each depth image to be evaluated.
[0071] In a possible implementation, after obtaining the image-level depth value evaluation results corresponding to each depth image to be evaluated, the computer device can determine the depth value evaluation results corresponding to the depth image sequence to be evaluated composed of the depth images to be evaluated.
[0072] In one possible implementation, taking into account that the depth map sequence to be evaluated may include multiple depth images to be evaluated corresponding to different target scenes, in order to obtain the depth value evaluation results corresponding to each scene, the computer device may group the depth images to be evaluated according to the target scenes corresponding to each depth image to be evaluated, and obtain image grouping results, wherein the depth images to be evaluated corresponding to the same target scene in the image grouping results are grouped together, thereby determining the scene-level depth value evaluation results corresponding to each group of depth images to be evaluated under different target scenes based on the image-level depth value evaluation results corresponding to each depth image to be evaluated and the image grouping results, and determining the depth value evaluation results of the depth map sequence to be evaluated based on the scene-level depth value evaluation results corresponding to each group of depth images to be evaluated.
[0073] Optionally, the image-level depth value evaluation result can be the evaluation score of each depth image to be evaluated at each visual feature level, the scene-level depth value evaluation result can be the average evaluation score of all image frames in each scene at each visual feature level, and the depth value evaluation result of the depth image sequence to be evaluated (i.e., the overall depth value evaluation result) can be the average evaluation score of all scenes at each visual feature level. Therefore, by dividing the depth value evaluation results into three different levels, depth value evaluation at different levels can be achieved, and an image frame with depth estimation problems can be quickly located.
[0074] In one possible implementation, a computer device can determine the visual features to be optimized based on the depth value evaluation results corresponding to the depth image sequence to be evaluated, wherein the depth value evaluation results corresponding to the visual features to be optimized are lower than a threshold, for example, the depth value measurement scores corresponding to the visual features to be optimized are lower than a threshold. Then, the computer device can quickly locate the target scene to be optimized corresponding to the visual features to be optimized based on the image grouping results, and determine the depth image to be optimized from the depth images to be evaluated corresponding to the target scene to be optimized based on the visual features to be optimized.
[0075] Step 304 , based on the depth value evaluation results corresponding to the depth map sequence to be evaluated, a solution evaluation result of the depth estimation solution to be evaluated is determined, where the solution evaluation result represents the depth estimation accuracy of the depth estimation solution to be evaluated.
[0076] The specific implementation of step 304 may refer to step 203, and this embodiment will not be described in detail here.
[0077] In the above embodiment, the depth value of each depth image to be evaluated is evaluated based on four different visual features, namely, foreground character outline, foreground object outline, regional smoothness and temporal stability, to obtain depth value evaluation results corresponding to different visual features, thereby achieving a more systematic and comprehensive objective quantitative evaluation of each frame of depth image, and improving the accuracy of depth value evaluation.
[0078] In addition, by dividing the depth image sequence to be evaluated into three different levels: overall, scene, and image, and determining the depth value evaluation results corresponding to different levels, it is helpful to quickly locate image frames with obvious depth estimation errors, thereby improving the efficiency of depth value evaluation.
[0079] In a possible implementation, for the same group of multi-view color image sequences, the computer device can also obtain depth map sequences to be evaluated corresponding to different depth estimation schemes, so that by evaluating each group of depth map sequences to be evaluated, the scheme evaluation results corresponding to each depth estimation scheme can be obtained, and then the depth estimation accuracy of each depth estimation scheme can be compared according to the scheme evaluation results corresponding to each depth estimation scheme.
[0080] In one possible implementation, for the true value depth map sequence generated based on the virtual engine, taking into account that there may be multiple version iteration processes for the same virtual scene in the virtual engine, that is, the same virtual scene may correspond to multiple groups of color image sequences, in order to determine the depth value optimization problems that may exist in the version iteration process, the computer device can perform depth value evaluation on the depth map sequence to be evaluated corresponding to each group of color image sequences, and obtain the depth value changes of pixel points in the color image corresponding to the same virtual scene during the version iteration process, which is conducive to developers to make more intuitive comparisons and make corresponding adjustments.
[0081] Please refer to Figure 5 , which shows a structural block diagram of an evaluation device for a depth estimation solution provided by an exemplary embodiment of the present application, the device comprising:
[0082] A first acquisition module 501 is used to acquire a depth image sequence to be evaluated, wherein the depth image sequence to be evaluated includes a plurality of depth images to be evaluated at different viewing angles, wherein the depth images to be evaluated correspond one-to-one to color images in a multi-view color image sequence, wherein the color images are obtained by photographing a target scene at different viewing angles, and wherein the depth images to be evaluated are obtained by performing depth estimation on the color images based on a depth estimation scheme to be evaluated;
[0083] An evaluation module 502 is used to perform depth value evaluation on the depth map sequence to be evaluated from a visual feature level based on the true value depth map sequence and the annotated multi-view color image sequence, and obtain a depth value evaluation result of the depth map sequence to be evaluated, wherein the true value depth image in the true value depth map sequence corresponds one-to-one to the depth image to be evaluated in the depth map sequence to be evaluated;
[0084] The result determination module 503 is used to determine the scheme evaluation result of the depth estimation scheme to be evaluated based on the depth value evaluation result corresponding to the depth map sequence to be evaluated, and the scheme evaluation result represents the depth estimation accuracy of the depth estimation scheme to be evaluated.
[0085] Optionally, the evaluation module 502 includes:
[0086] An evaluation unit is used to perform depth value evaluation on each depth image to be evaluated in the depth image sequence to be evaluated from the visual feature level based on the true value depth map sequence and the annotated multi-view color image sequence, and determine an image-level depth value evaluation result corresponding to each depth image to be evaluated, wherein the image-level depth value evaluation result represents the depth estimation of the depth image to be evaluated at the visual feature level, and the visual feature includes at least one of edge quality, regional smoothness and temporal stability;
[0087] The result determination unit is used to determine the depth value evaluation result of the depth image sequence to be evaluated based on the image-level depth value evaluation results corresponding to each depth image to be evaluated.
[0088] Optionally, the visual feature is the edge quality, and the edge quality includes a foreground person contour quality and a foreground object contour quality;
[0089] The evaluation unit is used for:
[0090] Based on the true value depth map sequence and the multi-view color image sequence annotated with character features, depth value evaluation is performed on each depth image to be evaluated, and a first image-level depth value evaluation result corresponding to each depth image to be evaluated in terms of the foreground character contour quality is determined;
[0091] Based on the true depth map sequence and the multi-view color image sequence annotated with object features, a depth value evaluation is performed on each depth image to be evaluated, and a second image-level depth value evaluation result corresponding to the foreground object contour quality of each depth image to be evaluated is determined.
[0092] Optionally, the visual feature is the smoothness of the region, and the evaluation unit is further used for:
[0093] Based on the true depth map sequence and the multi-view color image sequence annotated with regional features, a depth value evaluation is performed on each depth image to be evaluated, and a third image-level depth value evaluation result corresponding to the smoothness of the region of each depth image to be evaluated is determined.
[0094] Optionally, the visual feature is the temporal stability, and the evaluation unit is further used for:
[0095] Based on the true depth map sequence and the multi-view color image sequence annotated with temporal features, a depth value evaluation is performed on each depth image to be evaluated, and a fourth image-level depth value evaluation result corresponding to each depth image to be evaluated in terms of the temporal stability is determined.
[0096] Optionally, the result determination module 503 is used to:
[0097] Based on the target scene corresponding to each depth image to be evaluated, the depth images to be evaluated are grouped to obtain an image grouping result, wherein the depth images to be evaluated corresponding to the same target scene in the image grouping result are grouped as one group;
[0098] Determine scene-level depth value evaluation results corresponding to each group of depth images to be evaluated under different target scenes based on the image-level depth value evaluation results corresponding to each depth image to be evaluated and the image grouping results;
[0099] Based on the scene-level depth value evaluation results corresponding to each group of depth images to be evaluated, the depth value evaluation results of the depth image sequence to be evaluated are determined.
[0100] Optionally, the device further comprises:
[0101] A feature determination module, configured to determine a visual feature to be optimized based on the depth value evaluation result corresponding to the depth map sequence to be evaluated, wherein the depth value evaluation result corresponding to the visual feature to be optimized is lower than a threshold;
[0102] A scene determination module, used for determining the target scene to be optimized corresponding to the visual feature to be optimized based on the image grouping result;
[0103] The image determination module is used to determine the depth image to be optimized from the depth images to be evaluated corresponding to the target scene to be optimized based on the visual features to be optimized.
[0104] Optionally, the device further comprises:
[0105] A second acquisition module is used to acquire a multi-view color image sequence and a video image sequence corresponding to the target scene, wherein the video image sequence is obtained by shooting left and right videos of the target scene along a large baseline, and the target scene is kept in the center of the image during the shooting process;
[0106] A posture determination module, used to determine the camera posture corresponding to each color image in the multi-view color image sequence based on the multi-view color image sequence and the video image sequence;
[0107] The depth estimation module is used to obtain a true depth map sequence corresponding to the multi-view color image sequence through multi-view depth estimation based on the multi-view color image sequence and the camera pose corresponding to each color image.
[0108] To summarize, in an embodiment of the present application, in order to evaluate the depth estimation accuracy of the depth estimation scheme to be evaluated, first, a depth map sequence to be evaluated is obtained, and the depth map sequence to be evaluated is obtained by performing depth estimation on each color image in a multi-view color image sequence based on the depth estimation scheme to be evaluated, wherein the color image is obtained by photographing the target scene at different view angles, thereby performing depth value evaluation on the depth map sequence to be evaluated from the visual feature level according to the true value depth map sequence and the labeled multi-view color image sequence, that is, obtaining the depth value evaluation result of the depth map sequence to be evaluated, and then determining the scheme evaluation result of the depth estimation scheme to be evaluated according to the depth value evaluation result corresponding to the depth map sequence to be evaluated; adopting the scheme provided in the embodiment of the present application, the depth value evaluation of the depth map sequence to be evaluated is performed from the visual feature level through the true value depth image sequence and the labeled multi-view color image sequence, thereby improving the evaluation accuracy of the depth estimation scheme.
[0109] It should be noted that: the device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the implementation process thereof is detailed in the method embodiment, which will not be repeated here.
[0110] Figure 6 The block diagram of the structure of a computer device provided by an exemplary embodiment of the present application is shown. The computer device 600 may include one or more of the following components: a processor 610 and a memory 620 .
[0111] The processor 610 may include one or more processing cores. The processor 610 uses various interfaces and lines to connect the various parts of the entire computer device 600, and executes various functions and processes data of the computer device 600 by running or executing instructions, programs, code sets or instruction sets stored in the memory 620, and calling data stored in the memory 620. Optionally, the processor 610 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 610 can integrate one or more combinations of a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the touch display; the NPU is used to implement artificial intelligence (AI) functions; and the modem is used to process wireless communications. It is understandable that the above-mentioned modem may not be integrated into the processor 610, but may be implemented by a separate chip.
[0112] The memory 620 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 620 includes a non-transitory computer-readable storage medium. The memory 620 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 620 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data created according to the use of the computer device 600, etc.
[0113] In addition, those skilled in the art can understand that the structure of the computer device 600 shown in the above drawings does not constitute a limitation on the terminal, and the terminal may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, the computer device 600 also includes a display screen, a camera assembly, a microphone, a speaker, a radio frequency circuit, a sensor, an audio circuit, a WiFi module, a power supply, a Bluetooth module and other components, which will not be described in detail here.
[0114] An embodiment of the present application also provides a computer-readable storage medium, which stores at least one program, and the at least one program is loaded and executed by a processor to implement the evaluation method of the depth estimation scheme described in the above embodiments.
[0115] According to one aspect of the present application, a computer program product is provided, the computer program product including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a terminal reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the terminal executes the evaluation method of the depth estimation scheme provided in various optional implementations of the above aspects.
[0116] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable storage medium or transmitted as one or more instructions or codes on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.
[0117] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.< / label> < / label>
Claims
1. A method for evaluating a depth estimation scheme, characterized in that: The method comprises: Acquire a depth map sequence to be evaluated, wherein the depth map sequence to be evaluated includes a plurality of depth images to be evaluated at different viewing angles, the depth images to be evaluated correspond one-to-one to color images in a multi-view color image sequence, the color images are obtained by photographing a target scene at different viewing angles, and the depth images to be evaluated are obtained by performing depth estimation on the color images based on a depth estimation scheme to be evaluated; Based on the true value depth map sequence and the annotated multi-view color image sequence, a depth value evaluation is performed on the depth map sequence to be evaluated from the visual feature level to obtain a depth value evaluation result of the depth map sequence to be evaluated, wherein the true value depth images in the true value depth map sequence correspond one-to-one to the depth images to be evaluated in the depth map sequence to be evaluated; Based on the depth value evaluation results corresponding to the depth map sequence to be evaluated, a scheme evaluation result of the depth estimation scheme to be evaluated is determined, and the scheme evaluation result represents the depth estimation accuracy of the depth estimation scheme to be evaluated.
2. The method according to claim 1, characterized in that The method of performing depth value evaluation on the depth map sequence to be evaluated from the visual feature level based on the true value depth map sequence and the annotated multi-view color image sequence to obtain the depth value evaluation result of the depth map sequence to be evaluated includes: Based on the true value depth map sequence and the annotated multi-view color image sequence, perform depth value evaluation on each depth image to be evaluated in the depth map sequence to be evaluated from the visual feature level, and determine the image-level depth value evaluation result corresponding to each depth image to be evaluated, wherein the image-level depth value evaluation result represents the depth estimation of the depth image to be evaluated at the visual feature level, and the visual feature includes at least one of edge quality, regional smoothness and temporal stability; Based on the image-level depth value evaluation results corresponding to each depth image to be evaluated, the depth value evaluation result of the depth image sequence to be evaluated is determined.
3. The method according to claim 2, characterized in that The visual feature is the edge quality, and the edge quality includes the foreground character contour quality and the foreground object contour quality; The method of evaluating the depth value of each depth image to be evaluated in the depth image sequence to be evaluated based on the true value depth map sequence and the labeled multi-view color image sequence from the visual feature level, and determining the image-level depth value evaluation result corresponding to each depth image to be evaluated, includes: Based on the true value depth map sequence and the multi-view color image sequence annotated with character features, depth value evaluation is performed on each depth image to be evaluated, and a first image-level depth value evaluation result corresponding to each depth image to be evaluated in terms of the foreground character contour quality is determined; Based on the true depth map sequence and the multi-view color image sequence annotated with object features, a depth value evaluation is performed on each depth image to be evaluated, and a second image-level depth value evaluation result corresponding to the foreground object contour quality of each depth image to be evaluated is determined.
4. The method according to claim 2, characterized in that: The visual feature is the regional smoothness, and the method based on the true value depth map sequence and the labeled multi-view color image sequence, performs depth value evaluation on each depth image to be evaluated in the depth map sequence to be evaluated from the visual feature level, and determines the image-level depth value evaluation result corresponding to each depth image to be evaluated, and further includes: Based on the true depth map sequence and the multi-view color image sequence annotated with regional features, a depth value evaluation is performed on each depth image to be evaluated, and a third image-level depth value evaluation result corresponding to the smoothness of the region of each depth image to be evaluated is determined.
5. The method according to claim 2, characterized in that: The visual feature is the temporal stability, and the method based on the true value depth map sequence and the labeled multi-view color image sequence, performs depth value evaluation on each depth image to be evaluated in the depth map sequence to be evaluated from the visual feature level, and determines the image-level depth value evaluation result corresponding to each depth image to be evaluated, and further includes: Based on the true depth map sequence and the multi-view color image sequence annotated with temporal features, a depth value evaluation is performed on each depth image to be evaluated, and a fourth image-level depth value evaluation result corresponding to each depth image to be evaluated in terms of the temporal stability is determined.
6. The method according to claim 2, characterized in that The step of determining the depth value evaluation result of the depth image sequence to be evaluated based on the image-level depth value evaluation result corresponding to each depth image to be evaluated comprises: Based on the target scene corresponding to each depth image to be evaluated, the depth images to be evaluated are grouped to obtain an image grouping result, wherein the depth images to be evaluated corresponding to the same target scene in the image grouping result are grouped as one group; Determine scene-level depth value evaluation results corresponding to each group of depth images to be evaluated under different target scenes based on the image-level depth value evaluation results corresponding to each depth image to be evaluated and the image grouping results; Based on the scene-level depth value evaluation results corresponding to each group of depth images to be evaluated, the depth value evaluation results of the depth image sequence to be evaluated are determined.
7. The method according to claim 6, characterized in that The method further comprises: Determining a visual feature to be optimized based on the depth value evaluation result corresponding to the depth map sequence to be evaluated, wherein the depth value evaluation result corresponding to the visual feature to be optimized is lower than a threshold; Based on the image grouping result, determining the target scene to be optimized corresponding to the visual feature to be optimized; Based on the visual features to be optimized, a depth image to be optimized is determined from the depth images to be evaluated corresponding to the target scene to be optimized.
8. The method according to claim 1, characterized in that The method further comprises: Acquire a multi-view color image sequence and a video image sequence corresponding to the target scene, wherein the video image sequence is obtained by shooting left and right videos of the target scene along a large baseline, and the target scene is kept in the center of the image during the shooting process; Based on the multi-view color image sequence and the video image sequence, determining a camera pose corresponding to each color image in the multi-view color image sequence; Based on the multi-view color image sequence and the camera pose corresponding to each color image, a true depth map sequence corresponding to the multi-view color image sequence is obtained through multi-view depth estimation.
9. A depth estimation scheme evaluation device, characterized in that: The device comprises: A first acquisition module is used to acquire a depth map sequence to be evaluated, wherein the depth map sequence to be evaluated includes a plurality of depth images to be evaluated at different viewing angles, wherein the depth images to be evaluated correspond one-to-one to color images in a multi-view color image sequence, wherein the color images are obtained by photographing a target scene at different viewing angles, and wherein the depth images to be evaluated are obtained by performing depth estimation on the color images based on a depth estimation scheme to be evaluated; An evaluation module is used to perform depth value evaluation on the depth map sequence to be evaluated from the visual feature level based on the true value depth map sequence and the annotated multi-view color image sequence, and obtain the depth value evaluation result of the depth map sequence to be evaluated, wherein the true value depth image in the true value depth map sequence corresponds one-to-one to the depth image to be evaluated in the depth map sequence to be evaluated; The result determination module is used to determine the scheme evaluation result of the depth estimation scheme to be evaluated based on the depth value evaluation result corresponding to the depth map sequence to be evaluated, and the scheme evaluation result represents the depth estimation accuracy of the depth estimation scheme to be evaluated.
10. A computer device, characterized in that: The computer device includes a processor and a memory; the memory stores at least one instruction, and the at least one instruction is used to be executed by the processor to implement the evaluation method of the depth estimation solution as described in any one of claims 1 to 8.