Method, device, electronic device and storage medium for determining image depth information

By identifying feature regions and using a pre-trained model with guided filtering and nonlinear fusion, the method addresses misalignment and error accumulation in monocular depth estimation, enhancing accuracy and efficiency.

CN114066957BActive Publication Date: 2024-12-03BEIJING DAJIA INTERNET INFORMATION TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202010784818.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-08-06
Publication Date
2024-12-03
Estimated Expiration
2040-08-06

AI Technical Summary

Technical Problem

Existing methods for predicting depth information from monocular camera images face challenges due to misalignment of moving areas and errors accumulating in video frame images, leading to deviations in depth estimation.

Method used

A method involving determining feature regions with key points in historical frames, mapping depth values to target frames, and using a pre-trained depth prediction model with guided filtering and nonlinear fusion to improve accuracy.

Benefits of technology

This approach ensures accurate depth prediction by avoiding misalignment issues and enabling self-correction, resulting in higher precision and efficiency in depth estimation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114066957B_ABST
    Figure CN114066957B_ABST
Patent Text Reader

Abstract

The present disclosure is about a method, device, electronic device and storage medium for determining image depth information, the method comprising: obtaining the depth value of a pixel point in a feature area in a historical frame image, the historical frame image is located before a target frame image in a target video, and the target frame image contains a target area that matches the feature area; mapping the depth value to the target area according to the correspondence between the feature area and the target area, generating a depth value set corresponding to the target frame image, the depth value set recording the position information and depth value of the target point in the target area; determining the target depth map corresponding to the target frame image based on the color value of each pixel point in the target frame image and the above-mentioned depth value set. This method can avoid the problem of faults or difficulty in alignment in the motion area; and because it has self-correction capability in the depth prediction process, it has a high depth prediction accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision, and in particular to a method, device, electronic device and storage medium for determining image depth information. Background Art

[0002] Predicting the corresponding depth image based on a two-dimensional image taken by a monocular camera is an important topic in the field of computer vision. For video frame images, related technologies usually use the regression method of CNN (Convolutional Neural Networks) network to predict the depth of monocular images: first, use the CNN network to obtain the independent depth estimation results of the current frame image and the historical frame image, then project the historical frame to the current frame using the stereo matching method, and then fuse the aligned multiple frames to obtain the predicted depth map of the current frame image.

[0003] However, in the processing scenario of video frame images, since the photographer or the photographed object is often not stationary, the consecutive video frame images are not completely consistent. Therefore, there are often problems such as difficulty in aligning the moving areas in the historical frame images and the current frame images or there are faults in the boundaries of the fusion areas. In addition, the depth estimation results for subsequent video frames will cause serious deviations due to the accumulation of errors. Summary of the invention

[0004] The present disclosure provides a method, device, electronic device and storage medium for determining image depth information to at least solve the technical problems in the related art. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a method for determining image depth information is provided, comprising:

[0006] Determine a feature region including the key point and its adjacent pixels according to the key point in the historical frame image, and obtain a depth value of each pixel in the feature region, wherein the historical frame image is located before the target frame image in the target video, and the target frame image includes a target region matching the feature region;

[0007] Mapping the depth value of each pixel point to the target area according to the matching relationship between the feature area and the target area, generating a depth value set corresponding to the target frame image, wherein the depth value set records the position information and recorded depth values ​​of the target points in the target area, and the target points correspond to the each pixel point in the feature area;

[0008] Based on the color value of each pixel in the target frame image and the depth value set, a target depth map corresponding to the target frame image is determined.

[0009] Optionally, determining a feature area including the key point and its adjacent pixels according to the key point in the historical frame image includes:

[0010] Determine key points in the historical frame image;

[0011] An expansion area including the key point and its adjacent pixels is determined based on a predefined expansion rule, and the expansion area is determined as the feature area.

[0012] Optionally, the depth value of the target point is determined by:

[0013] Searching for a target area in the target frame image that matches the feature area;

[0014] Determine the pixel point corresponding to each target point in the target area in the feature area;

[0015] The depth value of the pixel point corresponding to the target point is determined as the depth value of the target point.

[0016] Optionally, determining a target depth map corresponding to the target frame image based on the color value of each pixel in the target frame image and the depth value set includes:

[0017] Determine the estimated depth value of each pixel point according to the color value of each pixel point in the target frame image;

[0018] Determining a target depth value of the target point according to the estimated depth value of the target point and the depth value set;

[0019] A target depth map corresponding to the target frame image is constructed based on the target depth value of the target point and the estimated depth value of the non-target point outside the target area in the target frame image.

[0020] Optionally, a difference between a target depth value of the target point and the recorded depth value is positively correlated with a difference between an estimated depth value of the target point and the recorded depth value.

[0021] Optionally, determining the estimated depth value of each pixel point according to the color value of each pixel point in the target frame image includes:

[0022] The color value of each pixel in the target frame image and the depth value set are input into a pre-trained depth prediction model, and an estimated depth map corresponding to the target frame image output by the depth prediction model is obtained, wherein the estimated depth map contains the estimated depth value of each pixel.

[0023] Optionally, the training method of the depth prediction model includes:

[0024] Determine a training sample including a color value of each pixel point in a sample target frame image and a sample depth value set, wherein the sample depth value set records position information and a sample depth value of the sample target point in the sample target frame image, and the sample depth value is a depth value of a pixel point matching the sample target point in a sample history frame image;

[0025] Inputting the training sample into the model to be trained to obtain a sample predicted depth map corresponding to the sample target frame image output by the model to be trained;

[0026] The model to be trained is trained according to the sample standard depth map and the sample predicted depth map to obtain the depth prediction model, wherein the sample standard depth map includes the sample actual depth map of the sample target frame image, or a fused depth map corresponding to the sample target frame image, and the fused depth map is obtained by nonlinear fusion of the sample actual depth map and the sample historical depth map corresponding to the sample historical frame image.

[0027] Optionally, the sample standard depth map is the fused depth map, and the difference between the fused depth value corresponding to the sample target point in the fused depth map and the sample historical depth value of the sample target point is positively correlated with the difference between the fused depth value and the actual depth value of the sample target point.

[0028] Optionally, random noise is added to the sample historical depth values ​​recorded in the sample depth value set.

[0029] Optionally, also include:

[0030] According to the color value of each pixel in the historical frame image, a guided filtering process is performed on the feature depth map of the historical frame image to filter out noise and / or texture of the feature depth map.

[0031] Optionally, the target frame image includes a first frame image of the target video, and the method further includes:

[0032] Obtaining estimated depth values ​​of all pixels in the target frame image according to the color values ​​of all pixels in the target frame image;

[0033] A depth map including the estimated depth values ​​of all the pixels is determined as the target depth map.

[0034] According to a second aspect of an embodiment of the present disclosure, a device for determining image depth information is provided, comprising:

[0035] a depth value acquisition module, configured to determine a feature area including the key point and its adjacent pixels according to the key point in the historical frame image, and to acquire the depth value of each pixel in the feature area, wherein the historical frame image is located before the target frame image in the target video, and the target frame image includes a target area matching the feature area;

[0036] A set generation module is configured to map the depth value of each pixel point to the target area according to the matching relationship between the feature area and the target area, and generate a depth value set corresponding to the target frame image, wherein the depth value set records the position information and recorded depth values ​​of the target points in the target area, and the target points correspond to the each pixel point in the feature area;

[0037] The first determination module is configured to determine a target depth map corresponding to the target frame image based on the color value of each pixel in the target frame image and the depth value set.

[0038] Optionally, the depth value acquisition module is further configured to:

[0039] Determine key points in the historical frame image;

[0040] An expansion area including the key point and its adjacent pixels is determined based on a predefined expansion rule, and the expansion area is determined as the feature area.

[0041] Optionally, the depth value acquisition module is further configured to:

[0042] Searching for a target area in the target frame image that matches the feature area;

[0043] Determine the pixel point corresponding to each target point in the target area in the feature area;

[0044] The depth value of the pixel point corresponding to the target point is determined as the depth value of the target point.

[0045] Optionally, the first determining module is further configured to:

[0046] Determine the estimated depth value of each pixel point according to the color value of each pixel point in the target frame image;

[0047] Determining a target depth value of the target point according to the estimated depth value of the target point and the depth value set;

[0048] A target depth map corresponding to the target frame image is constructed based on the target depth value of the target point and the estimated depth value of the non-target point outside the target area in the target frame image.

[0049] Optionally, a difference between a target depth value of the target point and the recorded depth value is positively correlated with a difference between an estimated depth value of the target point and the recorded depth value.

[0050] Optionally, the first determining module is further configured to:

[0051] The color value of each pixel in the target frame image and the depth value set are input into a pre-trained depth prediction model, and an estimated depth map corresponding to the target frame image output by the depth prediction model is obtained, wherein the estimated depth map contains the estimated depth value of each pixel.

[0052] Optionally, it also includes: a model training module configured to:

[0053] Determine a training sample including a color value of each pixel point in a sample target frame image and a sample depth value set, wherein the sample depth value set records position information and a sample depth value of the sample target point in the sample target frame image, and the sample depth value is a depth value of a pixel point matching the sample target point in a sample history frame image;

[0054] Inputting the training sample into the model to be trained to obtain a sample predicted depth map corresponding to the sample target frame image output by the model to be trained;

[0055] The model to be trained is trained according to the sample standard depth map and the sample predicted depth map to obtain the depth prediction model, wherein the sample standard depth map includes the sample actual depth map of the sample target frame image, or a fused depth map corresponding to the sample target frame image, and the fused depth map is obtained by nonlinear fusion of the sample actual depth map and the sample historical depth map corresponding to the sample historical frame image.

[0056] Optionally, the sample standard depth map is the fused depth map, and the difference between the fused depth value corresponding to the sample target point in the fused depth map and the sample historical depth value of the sample target point is positively correlated with the difference between the fused depth value and the actual depth value of the sample target point.

[0057] Optionally, random noise is added to the sample historical depth values ​​recorded in the sample depth value set.

[0058] Optionally, also include:

[0059] The guided filtering module is configured to perform guided filtering on the feature depth map of the historical frame image according to the color value of each pixel in the historical frame image, so as to filter out noise and / or texture of the feature depth map.

[0060] Optionally, the target frame image is a first frame image of the target video, and the device further includes:

[0061] A depth value estimation module is configured to obtain estimated depth values ​​of all pixels in the target frame image according to the color values ​​of all pixels in the target frame image;

[0062] The second determination module is configured to determine the depth map including the estimated depth values ​​of all the pixels as the target depth map.

[0063] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including:

[0064] processor;

[0065] a memory for storing instructions executable by the processor;

[0066] The processor is configured to execute the instructions to implement the method for determining image depth information as described in any embodiment of the first aspect above.

[0067] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is proposed. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the method for determining image depth information described in any embodiment of the first aspect above.

[0068] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0069] According to an embodiment of the present disclosure, by acquiring the depth value of the historical frame image and generating a set of depth values ​​corresponding to the target points matching the key points in the target frame image, the target points in the determined target depth map correspond one-to-one to the key points of the historical frame image, thereby avoiding the problem of difficulty in aligning the motion area or the existence of faults in the boundary of the fusion area; moreover, because the target depth value of the target point in the target frame image is determined based on the depth value of the key point and conforms to the corresponding law of nonlinear fusion, it has self-correction capability in the depth prediction process: the errors of the historical frame images will not cumulatively affect the depth prediction results of the target image, so a higher depth prediction accuracy can be achieved.

[0070] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0072] Figure 1 is a flow chart of a method for determining image depth information according to an embodiment of the present disclosure;

[0073] Figure 2 is a schematic diagram of a process of training a depth prediction model according to an embodiment of the present disclosure;

[0074] Figure 3 is a schematic diagram of a guided filtering process according to an embodiment of the present disclosure;

[0075] Figure 4 is a schematic diagram of a process of performing depth prediction using a depth prediction model according to an embodiment of the present disclosure;

[0076] Figure 5 is a schematic diagram of an image effect corresponding to a depth prediction process according to an embodiment of the present disclosure;

[0077] Figure 6 is a schematic diagram of a prediction result according to an embodiment of the present disclosure;

[0078] Figure 7 is a schematic block diagram of a device for determining image depth information according to an embodiment of the present disclosure;

[0079] Figure 8 is a structural diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0080] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0081] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0082] Predicting the corresponding depth image based on a two-dimensional image taken by a monocular camera is an important topic in the field of computer vision. For video frame images, related technologies usually use the regression method of the CNN network to predict the depth of the monocular image: first, the CNN network is used to obtain the independent depth estimation results of the current frame image and the historical frame image, and then the historical frame is projected to the current frame using the stereo matching method, and then the aligned multiple frames are fused to obtain the predicted depth map of the current frame image.

[0083] However, in the processing scenario of video frame images, since the photographer or the photographed object is often not stationary, the consecutive video frame images are not completely consistent. Therefore, there are often problems such as difficulty in aligning the moving areas in the historical frame images and the current frame images or there are faults in the boundaries of the fusion areas. In addition, the depth estimation results for subsequent video frames will cause serious deviations due to the accumulation of errors.

[0084] To this end, the present disclosure proposes a method for determining image depth information to solve the above-mentioned technical problems existing in the related art. Figure 1 is a flow chart of a method for determining image depth information according to an embodiment of the present disclosure; Figure 1 As shown, the method is applied to an image processing device, and the method may include:

[0085] Step 102, determining a feature area including the key point and its adjacent pixels based on the key point in the historical frame image, and obtaining the depth value of each pixel in the feature area, wherein the historical frame image is located before the target frame image in the target video, and the target frame image contains a target area matching the feature area.

[0086] The executor of the scheme disclosed herein may be an image processing device, for example, an electronic device running an image processing client corresponding to an image processing service, including but not limited to mobile phones, tablet computers, wearable devices, personal computers and other electronic devices; it may also be an electronic device running an image processing server corresponding to an image processing service, including but not limited to a physical server including an independent host, a virtual server or cloud server hosted by a host cluster, etc., and the present disclosure does not impose any restrictions on this.

[0087] Since the target video usually has multiple video frame images, the target frame image may be the first frame image or a non-first frame image. In one embodiment, when the target frame image is the first frame image of the target video, the estimated depth values ​​of all pixels in the target frame image can be obtained according to the color values ​​of all pixels in the target frame image; then the depth map containing the estimated depth values ​​of all pixels is determined as the target depth map. At this time, the estimated depth value is directly used as the target depth value of the first frame image, which avoids the unavailability of subsequent solutions caused by the absence of the previous frame image in the first frame image, and expands the scope of application of the disclosed solution. For example, the color space corresponding to the above color value can be an RGB color space. At this time, the corresponding color value includes the color value components corresponding to the three channels of R, G, and B. The depth estimation model constructed by the CNN network, CRF (conditional random field, conditional random field) algorithm, FastDepth algorithm, etc. can be used to calculate the estimated depth map corresponding to each pixel in the target frame image. The specific use of the depth estimation model can refer to the content disclosed in the relevant technology, which will not be repeated here.

[0088] In one embodiment, a feature extraction algorithm can be used to determine key points in the historical frame image, and then an extended area containing the key points can be determined based on a predefined extension rule, and the extended area can be determined as a feature area of ​​the historical frame image. At this time, the key points in the historical frame image are used as the starting point, and the extended area is determined according to the preset extension rule, thereby ensuring the regularity of the extended area determined in the historical frame image.

[0089] Furthermore, the depth value corresponding to the target point can be determined in the following manner: first search the target area in the target frame image that matches the feature area, then determine the pixel point corresponding to each target point in the target area in the feature area, and finally determine the depth value of the pixel point corresponding to the target point as the depth value of the target point.

[0090] Step 104: Map the depth value of each pixel point to the target area according to the matching relationship between the feature area and the target area, and generate a depth value set corresponding to the target frame image, wherein the depth value set records the position information and depth values ​​of the target points in the target area, and the target points correspond to the each pixel point in the feature area.

[0091] Because when actually predicting the image depth, the depth value contained in the feature depth map containing the historical frame image is often not a smooth and steady continuous value, but due to the presence of noise or surface texture of the photographed object, there is often a fluctuation in the depth value between adjacent pixels. Therefore, in one embodiment, the feature depth map of the historical frame image can be guided and filtered according to the color value of each pixel in the historical frame image to filter out the noise and / or texture of the feature depth map, thereby avoiding improving the accuracy of subsequent depth value prediction.

[0092] Step 106: Determine a target depth map corresponding to the target frame image based on the color value of each pixel in the target frame image and the depth value set.

[0093] In one embodiment, the estimated depth value of each pixel point can be determined based on the color value of each pixel point in the target frame image, and then the target depth value of the target point can be determined based on the estimated depth value of the target point and the depth value set, and finally a target depth map corresponding to the target frame image can be constructed based on the target depth value of the target point and the estimated depth value of the non-target point outside the target area in the target frame image. The constructed target depth map contains the target depth value of the target point and the estimated depth value of the non-target point, and the distribution scale of the target depth value and the estimated depth value is consistent with the distribution law of the target point and the non-target point in the target frame image, thereby ensuring the correspondence between the depth value and the pixel position.

[0094] Furthermore, the difference between the target depth value of the target point and the recorded depth value of the target point recorded in the depth value set is positively correlated with the difference between the estimated depth value of the target point and the recorded depth value. In other words, when the estimated depth value of the target point is closer to the recorded depth value, the target depth value of the target point is also closer to the recorded depth value; conversely, when the difference between the estimated depth value of the target point and the recorded depth value is greater, the difference between the target depth value of the target point and the recorded depth value is also greater. It can be seen that no matter how the relative size relationship between the estimated depth value and the recorded depth value of the target point changes, it can always ensure that the target depth value of the target point is close to the recorded depth value, and then closer to its true depth value.

[0095] Furthermore, a pre-trained depth prediction model can be used to predict the estimated depth value of each pixel, and then a corresponding estimated depth map can be constructed based on the estimated depth value of each pixel. For example, the color value and depth value set of each pixel in the target frame image can be input into a pre-trained depth prediction model, and the estimated depth map corresponding to the target frame image output by the depth prediction model can be obtained, and the estimated depth map contains the estimated depth values ​​of each pixel. Performing depth estimation on the video frame image of the target video using a pre-trained depth prediction model not only helps to improve the efficiency of depth estimation, reduce the processing time of video depth estimation and thus improve user experience, but also guarantees the prediction accuracy of depth prediction to a certain extent.

[0096] The above-mentioned depth prediction model can be trained in the following manner: first determine a training sample including the color value of each pixel in the sample target frame image and a sample depth value set, wherein the sample depth value set records the position information and sample depth value of the sample target point in the sample target frame image, and the sample depth value is the depth value of the pixel matching the sample target point in the sample history frame image, and then input the training sample into the model to be trained to obtain the sample prediction depth map corresponding to the sample target frame image output by the model to be trained, and finally train the model to be trained according to the sample standard depth map and the sample prediction depth map to obtain the depth prediction model, wherein the sample standard depth map is the sample actual depth map of the sample target frame image, or the fused depth map corresponding to the sample target frame image, and the fused depth map can be obtained by nonlinear fusion of the sample actual depth map and the sample history depth map corresponding to the sample history frame image. Wherein, after the model to be trained outputs a prediction result for any training sample, the model parameters can be adjusted according to the difference between the prediction result and the true value, or after the model to be trained outputs a prediction result for multiple training samples, the adjustment can be performed, that is, a suitable Batch can be selected for training to improve the training efficiency of the model. The depth estimation model is trained by using training samples containing a set of color values ​​and sample depth values ​​of each pixel in the sample target frame image, thereby ensuring that the training results of the depth estimation model are consistent with the actual usage scenario.

[0097] Among them, when the sample standard depth map is a fused depth map, the difference between the fused depth value corresponding to the sample target point in the fused depth map and the sample historical depth value of the sample target point can be positively correlated with the difference between the fused depth value and the actual depth value of the sample target point.

[0098] Furthermore, the above-mentioned sample target frame image and sample historical frame image can both be original frame images contained in the sample video, wherein the sample target actual depth map can include the real depth value of the sample target frame image, and the sample feature depth map can include the real depth value of the sample historical frame image; or, any frame image of the sample target frame image and the sample historical frame image can be the original image contained in the sample video, and accordingly, the depth map corresponding to any frame image includes the real depth value collected from any frame image, and the depth map corresponding to any frame image is used to simulate the depth map corresponding to another frame image.

[0099] In one embodiment, in order to improve the training accuracy of the depth prediction model, the training samples should be as close to the real data as possible. Therefore, random noise, such as salt and pepper noise, Gaussian noise, etc., can be added to the sample historical depth values ​​recorded in the sample depth value set to simulate sample data that is closer to the real data, thereby improving the training accuracy of the depth prediction model and thus improving the efficiency and accuracy of model training.

[0100] In one embodiment, the depth value set may also include the depth value of non-target points represented by preset meaningless numerical values. In this case, a regional mask map corresponding to the depth value set may also be generated, and the mask value in the regional mask map is used to characterize the position of the target point and / or non-target point in the target frame image; then, when the estimated depth value and the depth value set corresponding to the target point are input into the depth prediction model, the regional mask map is input into the depth prediction model, so that the depth prediction model distinguishes the depth values ​​corresponding to the target point and the non-target point in the depth value set based on the regional mask map. This ensures the accuracy of the depth prediction model in predicting the depth value of the target frame image.

[0101] According to an embodiment of the present disclosure, by acquiring the depth value of the historical frame image and generating a set of depth values ​​corresponding to the target points matching the key points in the target frame image, the target points in the determined target depth map correspond one-to-one to the key points of the historical frame image, thereby avoiding the problem of difficulty in aligning the motion area or the existence of faults in the boundary of the fusion area; moreover, because the target depth value of the target point in the target frame image is determined based on the depth value of the key point and conforms to the corresponding law of nonlinear fusion, it has self-correction capability in the depth prediction process: the errors of the historical frame images will not cumulatively affect the depth prediction results of the target frame image, so a higher depth prediction accuracy can be achieved.

[0102] The above depth prediction scheme can be implemented using a pre-trained depth prediction model. Figure 2 The schematic diagram of the depth prediction model training process shown in FIG. 1 describes the depth prediction model training process in detail. Figure 2 As shown, the process may include the following steps:

[0103] Step 201, obtaining a target sample frame image.

[0104] In one embodiment, a training sample may be determined before training the depth prediction model, and the training sample may include a target sample frame image and a historical sample frame image. As an exemplary embodiment, the target sample frame image is annotated with its corresponding actual depth map, and an image sampled by a binocular camera or a depth of field camera may be used as the target sample frame image, or an image collected or predicted by other methods may be used as the target sample frame image. In order to ensure the reliability of the sample and thus improve the efficiency of model training, regardless of the source of the target sample frame image, the reference depth map of the target sample frame image should be as accurate as possible, such as using the depth map collected by the binocular camera as the reference depth map of the target sample frame image, or using the depth map calculated by other methods and verified for accuracy as the reference depth map of the target sample frame image.

[0105] As another exemplary embodiment, the historical sample frame images are also annotated with their corresponding reference depth maps, and images sampled using a binocular camera or a depth-of-field camera can also be used as historical sample frame images. For example, the first frame image of two adjacent frames extracted from a video shot by a binocular camera is used as a historical sample frame image, and the second frame image is used as a target sample frame image. Of course, when the actual sample data is small, the historical sample frame image can be expanded by data augmentation based on the target sample frame image obtained by actual sampling. The specific process can be seen in the following steps 202-204 (i.e. Figure 2 steps in the dashed box).

[0106] Step 202: Estimate the estimated depth map of the target sample frame image by using the estimation model.

[0107] In this embodiment, the estimation model can be a CNN network model, a CRF model, a FastDepth model, etc. The specific model usage process can refer to the content disclosed in the relevant technology, which will not be repeated here. The estimated depth map obtained by estimation corresponds to the target sample frame image. For example, if the target sample frame image is an image of 1024×768 pixels, then its corresponding estimated depth map is also an image of 1024×768 pixels. The value corresponding to any pixel point of the estimated depth map is the estimated depth value corresponding to the pixel point. In fact, the "depth maps" involved in this specification are all depth maps corresponding to the corresponding images. The value corresponding to any pixel point of the depth map is the image depth value corresponding to the pixel point, for example, it can be a single-precision floating-point value.

[0108] Step 203 , performing random scale changes on the estimated depth map of the target sample frame image to obtain the estimated depth map of the historical sample frame image.

[0109] In one embodiment, in order to obtain more test samples based on fewer measured images, data augmentation processing can be performed on the target sample frame image: the target sample frame image is subjected to random scale change. There can be many specific scale change methods, for example, flip transformation, random crop, color jittering, translation transformation, scale transformation, contrast transformation, noise disturbance, rotation transformation / reflection transformation, etc. In fact, in the subsequent training process of this embodiment, only the estimated depth map of the target sample frame image is used. Therefore, in order to reduce the workload of scale change, this step 203 can only perform the above-mentioned scale change processing on the estimated depth map of the target sample frame image, thereby realizing the augmentation of the estimated depth map, and the new estimated depth map obtained after the augmentation processing can be used as the estimated depth map of the historical sample frame image.

[0110] Step 204 : Perform guided filtering on the estimated depth map of the historical sample frame image to obtain a relatively clearer estimated depth map.

[0111] In fact, the estimated depth map of the historical sample frame image obtained by the above scale change processing may contain unstable factors such as noise and texture, which is not conducive to subsequent processing and related calculations. Therefore, in one embodiment, the estimated depth map of the historical sample frame image can be processed by guided filtering to eliminate the noise and / or texture of the estimated depth map. The specific process can be seen in Figure 3 As shown in the schematic diagram, the estimated depth map of the historical sample frame image to be processed can be used as the input image P, and the preset guide image I can be input. The output image Q obtained by the guided filtering process is a clearer estimated depth map. For the convenience of subsequent description, the clearer estimated depth map processed by the guided filtering is still referred to as the estimated depth map.

[0112] Step 205 : acquiring a sparse depth block corresponding to a feature area in a target sample frame image based on the estimated depth map corresponding to the historical sample frame image.

[0113] In one embodiment, a key point extraction algorithm may be first used to extract key points in the historical sample frame image. For example, an ORB (Oriented FAST and Rotated BRIEF, fast extraction and description algorithm) algorithm, a FAST algorithm, a BRIEF algorithm, a Susan algorithm, or a DoG algorithm may be used for extraction. The use process of the above algorithms can refer to the content disclosed in the relevant technology, and will not be repeated here. Specifically, the key points in the historical sample frame image may be the eyes, corners of the mouth, moles of the photographed object, paintings, nails, corners of the wall, fonts, stains on paper, etc. The specific form of the key points may be determined by the above algorithm or pre-set before using the model corresponding to the above algorithm, and the present disclosure does not limit this.

[0114] In fact, the above key point extraction process can simultaneously extract the feature area where the key point is located, such as the area of ​​a preset area around the key point, etc. The estimated depth value of each pixel in each feature area (there may be several feature areas) is obtained, that is, a sparse depth block corresponding to the feature area in the historical sample frame image is obtained, wherein any feature area corresponds to a sparse depth block, and different sparse depth blocks can be separated or connected to each other.

[0115] Step 206: Add random noise to the sparse depth block and obtain a sparse depth map (corresponding to the aforementioned depth value set).

[0116] In one embodiment, corresponding to the above steps 202-204, since the historical sample frame images obtained by data augmentation are images in an ideal state, in order to ensure that the model training results are as close to the actual situation as possible, random noise can be added to the depth value of each pixel in the sparse depth block. For example, random noise such as salt and pepper noise and Gaussian noise can be added to make the sparse depth block after adding noise closer to the actual scene to improve the accuracy of model training. Of course, noise can also be added to the sparse depth map corresponding to the target area obtained from other devices according to actual conditions. The present disclosure does not limit the timing of adding random noise.

[0117] In one embodiment, a feature matching algorithm may be used to search for a target area matching the coefficient depth block in the target sample frame image. For example, a feature matching algorithm such as sparse optical flow or FAST may be used for the search. Then, each sparse depth block is placed at the corresponding target area, so that each sparse depth block of the feature area is spliced ​​into a sparse depth map of the target area. In addition, to ensure the size integrity of the sparse depth map, the depth values ​​corresponding to non-target points in the sparse depth map may be set to meaningless values, such as uniformly set to "0".

[0118] Step 207, obtaining a region mask map.

[0119] In the case where the depth values ​​corresponding to non-target points in the sparse depth map are set to meaningless values, it is difficult for the depth prediction model to distinguish whether the above meaningless values ​​are really meaningless (e.g., the depth prediction model cannot distinguish whether the depth of the pixel points set to "0" in the sparse depth map is really zero). To solve this problem, in one embodiment, a feature matching algorithm can be used to search for a target area matching the sparse depth block in the target sample frame image, and then the mask map corresponding to each sparse depth block can be directly determined according to the position coordinates of each pixel point in the sparse depth block of the target sample frame image in the target sample frame image; or the mask map corresponding to each sparse depth block can be determined according to the position coordinates of each pixel point in the sparse depth map of the target area determined in step 206 in the target sample frame image. Among them, the mask value corresponding to the pixel point corresponding to the sparse depth block (i.e., corresponding to the feature area) in the determined mask map can be set to the first value, and the mask value corresponding to the other (i.e., corresponding to the non-feature area) pixel point can be set to the second value, and the subsequent depth prediction model can determine the feature area according to each mask value in the mask. For example, the mask values ​​corresponding to the pixels of the sparse depth block can be uniformly set to "1", and the mask values ​​corresponding to other pixels can be set to "0", so that the depth prediction model can determine the pixel points corresponding to the pixels with mask values ​​of "1" in the mask map in the target sample frame image as the target points based on the mask map.

[0120] In fact, the above-mentioned sparse depth map also contains the position correspondence between each diluted depth value and the pixel point in the corresponding target area. Therefore, this step is an optional step, that is, this step may not be performed, and the depth prediction model determines the target area according to the position correspondence contained in the sparse depth map after receiving it. The specific process will not be repeated here.

[0121] Step 208: Use the depth prediction model to predict a predicted depth map of the target sample frame image.

[0122] At this time, the obtained mask image is used to represent the position of the target point in the target sample frame image, and the sparse depth map is used to represent the depth value of the key point corresponding to each target point. Figure 3 The depth prediction model is combined with the input depth prediction model to obtain a predicted depth map of the target sample frame image output by the model. The depth value of any pixel in the predicted depth map is the depth prediction result of the pixel by the depth prediction model.

[0123] In fact, the above-mentioned target sample frame image can be the value of each pixel in the target sample frame image in a preset color space. For example, the color space corresponding to the above-mentioned color value can be an RGB color space. At this time, the corresponding color value includes the color value components corresponding to the three channels of R, G, and B; for another example, the color space corresponding to the above-mentioned color value can also be a CMY color space. At this time, the input to the depth prediction model is the color value component corresponding to the three channels of C, M, and Y of the target sample frame image. Of course, the above-mentioned color space can also be SHV, CMYK, CIEXYZ, YCC, YUV or others, and the present disclosure does not limit this. However, it should be noted that if the depth prediction model is trained using the color values ​​of the training samples in a certain color space, then when using the trained depth prediction model to predict the depth map of the target image, it is also necessary to use the color value of the target image in the color space accordingly.

[0124] Step 209 , nonlinearly fuse the estimated depth map of the historical sample frame image with the actual depth map of the target sample frame image to obtain a training target.

[0125] In one embodiment, different fusion methods may be used depending on the historical sample frame images. As an exemplary embodiment, when the historical sample frame image is an extended sample image obtained by data augmentation based on the target sample frame image, the estimated depth map of the historical sample frame image may be nonlinearly fused with the actual depth map of the target sample frame image, and the obtained fusion result may be used as the training target of the depth prediction model. As another exemplary embodiment, when the historical sample frame image is the original image actually collected, the real depth map of the collected historical sample frame image may be nonlinearly fused with the actual depth map of the current frame of the target sample frame image, and the obtained fusion result may be used as the training target of the depth prediction model.

[0126] A depth map usually includes depth values ​​corresponding to multiple pixels, so the above fusion process is actually a fusion of the depth values ​​corresponding to each pixel in turn. The above fusion process can be performed only on the depth value corresponding to a single pixel, or on multiple pixels at one time, such as all key points of the historical sample frame image, all pixels of the historical sample frame image, etc. The present disclosure does not limit this.

[0127] In one embodiment, taking the historical sample frame image as an extended sample image obtained by data augmentation based on the target sample frame image as an example, for any pixel point (the actual depth value of the corresponding target sample frame image is recorded as p0, and the estimated depth value of the corresponding historical sample frame image is recorded as p1), the above fusion can be performed based on the following formula (1):

[0128] Target = p1 + (p0 - p1)*r (1)

[0129] Among them, r = f(|p0-p1|), f(x) = 1 / (1-exp(ax)), and a is a preset constant. It can be seen from the expression of the f(x) function that when x is small, f(x) is close to 0, and when x is large, f(x) is close to 1, and f(x) is a continuous function. Therefore, the smaller the deviation between p1 and p0, the closer the output training target Target is to p1; the larger the deviation between p1 and p0, the closer the output training target Target is to p0. In other words, the closer the estimated depth value of the historical sample frame image is to the actual depth value of the target sample frame image, the closer the training target Target is to the estimated depth value of the historical sample frame image (that is, close to the actual depth value of the target sample frame image); and the farther the estimated depth value of the historical sample frame image is from the actual depth value of the target sample frame image, the closer the training target Target is to the actual depth value of the target sample frame image.

[0130] Step 210 , calculating the deviation between the predicted depth map of the target sample frame image and the training target, and adjusting the model parameters of the depth prediction model based on the deviation.

[0131] Based on the deviation loss between the predicted depth map of the target sample frame image output by the above-mentioned depth prediction model and the training target (in fact, it is the deviation between the predicted depth value corresponding to a pixel point in the target sample frame image and the training target corresponding to the key point matching the point), the model parameters of the depth prediction model can be adjusted to make the output of the depth prediction model as close as possible to the actual depth map of the target sample frame image. For example, one or more of the network layers, convolution kernel size, sliding step size, optimization algorithm, activation function, Dropout layer, mini-batch value, learning rate, etc. of the depth prediction model can be adjusted, and the specific adjustment process will not be repeated.

[0132] In one embodiment, the deviation between the predicted depth value and the training target may be measured by the RMSE (Root Mean Square Error) of the two.

[0133] For example, in order to quantitatively describe the accuracy of depth estimation, for n pixels, the RMSE shown in the following formula (2) can be used as the evaluation index, where p i Represents the true depth value of any pixel i, y i Represents the predicted depth value of any pixel i. The smaller the RMSE, the higher the accuracy of the depth estimation.

[0134]

[0135] For another example, in order to quantitatively describe the stability of depth estimation, the RMSE shown in the following formula (3) can be used as the evaluation index. For the depth maps Y1 and Y2 corresponding to the continuous historical sample frame images and the target sample frame images, Y 1i represents the predicted depth value of Y1 at any pixel i, Y 2i Represents the predicted depth value of Y2 at any pixel position i. The smaller the RMSE, the smaller the jitter of Y1 and Y2, that is, the more stable the depth prediction.

[0136]

[0137] For an entire target video, the average value of the RMSE error of the depth maps corresponding to all two consecutive frames in the video can be calculated as an evaluation indicator of the stability of the depth estimation of the target video.

[0138] In fact, because the above formula (2) needs to know the actual depth map, and the formula (3) only needs to know the predicted depth value, the above formula (2) is only applicable to the model prediction stage, and the formula (3) is not only applicable to the model prediction stage, but also can be used to evaluate the prediction effect of the model in the actual use stage of the model. Of course, the deviation between the predicted depth map of the target sample frame image output by the above-mentioned depth prediction model and the training target can also be evaluated using other indicators besides RMSE, and the present disclosure does not limit this. In one embodiment, the maximum threshold of the deviation loss can be pre-set, and the threshold can be used as the basis for stopping training: when a certain output of the depth prediction model or the output of a preset number of consecutive times is not greater than the maximum threshold of the deviation loss, the training process of the depth prediction model can be stopped. After stopping, the depth prediction model can be used directly for depth prediction, or the model can be tested and the test can be stopped when the preset test conditions are met, thereby completing the training process of the depth prediction model.

[0139] The model training process is the process of minimizing the deviation loss as much as possible by adjusting the model parameters of the depth prediction model, so that the predicted depth map of the target sample frame image output by the model is as close to the training target Target as possible. From the above analysis of formula (1), it can be seen that no matter whether the estimated depth value of the historical sample frame image is close enough to the actual depth value of the target sample frame image, the training target Target is always close to the actual depth value of the target sample frame image, thereby ensuring that the predicted depth output by the above-trained depth prediction model for the target image is close to the actual depth map of the target image when used, thereby ensuring the accuracy of the depth prediction model.

[0140] At this point, the training process of the depth prediction model is completed, and the model can be used to realize depth estimation of images. In the following, the target frame image is taken as the current frame image in the target video (any video frame image in the target video), and the historical frame image is taken as the previous frame image in the target video whose playback sequence is before the current frame image. Figure 4-Figure 7 The process of using the trained depth prediction model to predict the depth of the current frame image is described in detail. Figure 4 A schematic diagram of a process of performing depth prediction using a depth prediction model is shown, and the above prediction process may include the following steps:

[0141] Step 401, determining the current frame image in the target video.

[0142] In this embodiment, any video frame image in the target video to be processed can be determined as the current frame image. Before determining the current frame image, the target video can be firstly subjected to video frame image extraction, that is, a plurality of continuous video frame images are extracted from the target video. Among the continuous video frame images corresponding to the target video, the first frame image or the non-first frame image can be determined as the current frame image. Because there is no corresponding previous frame image for the first frame image, when the current frame image is the first frame image of the target video, the estimated depth values ​​of all pixels in the target frame image can be obtained according to the color values ​​of all pixels in the target frame image, and then the depth map containing the estimated depth values ​​of all pixels is determined as the target depth map. The color space where the above color values ​​are located can be the same as the preset color space where the color values ​​of the samples used in the depth prediction model training stage are located to ensure the accuracy of the model prediction. When the current frame image is a non-first frame image of the target video, the process can proceed to step 402, and the depth prediction model can be used to perform depth prediction on the current frame image through subsequent steps.

[0143] Step 402: Obtain a predicted depth map of the previous frame of image.

[0144] Because the current frame image is not the first frame image of the target video, there is a previous frame image of the current frame, and the depth prediction of the video frame image of the target video is performed on each frame image in sequence. At this time, the predicted depth map of the previous frame image that has completed the depth prediction can be obtained. The corresponding value of any pixel point in the previous frame image in the predicted depth map is the predicted depth value corresponding to the pixel point. When the above model training is completed, the above predicted depth value should be close to the actual depth value corresponding to the pixel point (the error between the settings meets the error tolerance threshold of the depth prediction network).

[0145] Step 403: Perform guided filtering on the predicted depth map of the previous frame image.

[0146] In the actual prediction process, the predicted depth map of the previous frame image may contain unstable factors such as noise and texture, which is not conducive to subsequent processing and related calculations. Therefore, in one embodiment, the predicted depth map containing the depth value can be guided by filtering based on the color value of the previous frame image to filter out the noise and / or texture of the predicted depth map to obtain a clearer predicted depth map. For the specific process, see Figure 3 The schematic diagram of the guided filtering process is shown in FIG.

[0147] Step 404: Obtain a sparse depth block of the previous frame image.

[0148] In one embodiment, a key point extraction algorithm may be used to extract key points in a historical frame image, and then an extended area containing key points may be determined based on a predefined extension rule, and the extended area may be determined as a feature area. For example, an ORB algorithm, a FAST algorithm, a BRIEF algorithm, a Susan algorithm, or a DoG algorithm may be used for extraction. The specific use process of the model corresponding to the above algorithm may refer to the content disclosed in the related art, and will not be repeated here. Specifically, the key points in the historical sample frame image may be the eyes, corners of the mouth, moles of the photographed object, paintings, nails, corners of the wall, fonts, stains on paper, etc. At this time, the key point is located in the extended area, and the area composed of the predicted depth values ​​corresponding to the feature area may be determined as a sparse depth block. A sparse depth block is composed of one or more feature areas, and the depth value corresponding to any pixel in the sparse depth block is the predicted depth value of the key point corresponding to the pixel.

[0149] See also Figure 5 A schematic diagram of an image effect corresponding to the depth prediction process shown. Figure 5 (a) shows the previous frame of the target image. Figure 5 (a) includes a wall painting 501a and a wall corner 502a; Figure 5 (f) shows the current frame image in the target image. Similarly, Figure 5 (h) includes the painting 501h on the wall and the corner 502h. It can be seen that the position of the subject in the current frame image is different from that in the previous frame image. The position of the camera in the current frame image is closer to the corner. This may be because the photographer moves, causing the lens position to change during the shooting process. Assume that the predicted depth map of the previous frame image obtained in step 402 (or after the guided filtering in step 403) is as follows Figure 5 As shown in (b), the depth value of any pixel point 501b is the predicted depth value corresponding to the point.

[0150] for Figure 5For the previous frame image shown in (a), a key point extraction algorithm can be used to extract key points corresponding to the four corners and wall corners of the painting area in the previous frame image, and then the extended area corresponding to the key points is determined based on the preset extension rules. For example, the extension rule for the two-side angle (corresponding to the four corners of the painting) can be preset to take the two-side angle as the center point of a rectangular area with a length of a and a width of b, and the corresponding rectangular area is the extended area corresponding to the two-side angle; for another example, the extension rule for the three-side angle (corresponding to the wall corner) can be preset to take the three-side angle as the center of a circular area with a radius of r, and the corresponding circular area is the extended area corresponding to the three-side angle. The above-mentioned side lengths a, b and radius r can be absolute length values, such as length values ​​in units of centimeters, inches, etc., or relative length values, such as length values ​​in units of the number of pixels, and the present disclosure is not limited to this. Determined in the above manner Figure 5 The extended areas corresponding to the previous frame image shown in (a) are as follows Figure 5 (c) includes a rectangular expansion area 501c and a circular expansion area 502c. Each of the above expansion areas contains the above key points. For example, the circular expansion area 502c may include key points (pixel points corresponding to the center of the corner) and non-key points (pixel points corresponding to the three walls in the circular area). Figure 5 (b) and Figure 5 (d)- Figure 5 In (f), for the sake of simplicity, non-key points are not shown (all four pixels in the circular extended area 502c are described as key points), and each extended area together constitutes a feature area corresponding to the previous frame image.

[0151] Combined with Figure 5 (b) The predicted depth map of the previous frame and Figure 5 The characteristic area shown in (c) can be obtained as follows Figure 5 The sparse depth blocks 501d shown in (d) together constitute a sparse depth map of the previous frame image.

[0152] Step 405: determine the target area in the current frame image by matching the feature area.

[0153] In one embodiment, a feature matching algorithm can be used to search for a target area matching the extended area in the current frame image, and the pixel points in the searched target area are the aforementioned target points. For example, a feature matching algorithm such as sparse optical flow and FAST disclosed in the relevant technology can be used to search for the target area, and the specific search process will not be repeated. For the corresponding search results, see Figure 5 As shown in (g), it includes a rectangular target area 501g and a circular target area 502g.

[0154] Step 406: Obtain a sparse depth map corresponding to the target area.

[0155] Based on the above Figure 5 (d) shows the sparse depth block of the previous frame and Figure 5 (g) shows the target area of ​​the current feature image, and the sparse depth map corresponding to the target area can be obtained. For example, each sparse depth block can be placed in the corresponding target area, so that each sparse depth block of the feature area is spliced ​​into a sparse depth map corresponding to the target area. In addition, to ensure the size integrity of the sparse depth map, the depth values ​​corresponding to non-target points in the sparse depth map can be set to meaningless values, such as uniformly set to "0". See Figure 5 (f) The value of the target point corresponding to the target area in the sparse depth map is the predicted depth value 501f of the corresponding key point, and the value of the non-target point corresponding to the non-target area is the preset meaningless value "0"; of course, the value of the non-target point may not be set to a meaningless value.

[0156] Step 407: Obtain a mask image corresponding to the sparse depth image.

[0157] In the case where the depth values ​​corresponding to non-target points in the sparse depth map are set to meaningless values, it is difficult for the depth prediction model to distinguish whether the meaningless values ​​are really meaningless (for example, the depth prediction model cannot distinguish whether the depth of the pixel set to "0" in the sparse depth map is really zero). To solve this problem, in one embodiment, a mask map corresponding to the depth value set can be generated, and the mask values ​​in the mask map are used to represent the positions of the target points and / or non-target points in the current frame image. The above mask map is as follows: Figure 5 (e), wherein the mask value of the target point 502e corresponding to the sparse depth block is uniformly set to "1", and the mask value corresponding to the non-target point 501e is set to "0", so that the depth prediction model determines the specific position of the target point with a mask value of "1" in the mask map in the current frame image based on the mask map. Currently, the masks corresponding to the above-mentioned target points and non-target points can also be other preset values, which will not be repeated.

[0158] In fact, the above-mentioned sparse depth map also contains the position correspondence between each diluted depth value and the pixel point in the corresponding target area. Therefore, this step is an optional step, that is, this step may not be performed, and the depth prediction model determines the target point according to the position correspondence contained in the sparse depth map after receiving it. The specific process will not be repeated here.

[0159] Step 408: The values ​​of each pixel of the current frame image in the color space, the sparse depth map and the mask map are combined and input into the depth prediction model to obtain the predicted depth map of the current frame image output by the depth determination model.

[0160] At this time, the obtained mask image is used to represent the position of the target point in the current frame image, and the sparse depth map is used to represent the predicted depth value of the key point corresponding to each target point. At this time, the values ​​of each pixel in the current frame image in the preset color space, the sparse depth map and the mask can be Figure 3 The depth prediction model is combined with the input depth prediction model to obtain the current frame predicted depth map of the current frame image output by the model. The depth value of any pixel in the current frame predicted depth map is the result of the depth prediction model performing depth prediction on the pixel, and the depth value can be a floating point value.

[0161] like Figure 6 In the current frame predicted depth map corresponding to the current frame image shown, the predicted depth value corresponding to any pixel point 601 is a floating-point value with the unit of "meter (m)". Currently, other forms of current frame predicted depth maps can also be output according to the model settings and actual conditions, which will not be repeated here.

[0162] Because the predicted depth value output by the depth prediction model trained as above for the target frame image can be as close as possible to the actual depth value of the target frame image, it can ensure that the current frame predicted depth map output by the depth prediction model is close to the actual depth map of the current frame image, and ensure the prediction accuracy of the depth prediction model.

[0163] Corresponding to the above-mentioned embodiment of the method for determining image depth information, the present disclosure also proposes an embodiment of an apparatus for determining image depth information.

[0164] Figure 7 It is a schematic block diagram of a device for determining image depth information according to an embodiment of the present disclosure. The device for determining image depth information shown in this embodiment can be applicable to image processing applications, and the application is applicable to devices running image processing clients corresponding to image processing services, including but not limited to mobile phones, tablet computers, wearable devices, personal computers and other electronic devices; it can also be a device running image processing servers corresponding to image processing services, including but not limited to a physical server including an independent host, a virtual server or cloud server hosted by a host cluster, etc.; the image can be a video frame image, or a plurality of images taken in succession, or an image in other forms, and the present disclosure does not limit this.

[0165] like Figure 7 As shown, the device for determining image depth information may include:

[0166] The depth value acquisition module 701 is configured to determine a feature area including the key point and its adjacent pixels according to the key point in the historical frame image, and acquire the depth value of each pixel in the feature area, wherein the historical frame image is located before the target frame image in the target video, and the target frame image includes a target area matching the feature area;

[0167] A set generation module 702 is configured to map the depth value of each pixel point to the target area according to the matching relationship between the feature area and the target area, and generate a depth value set corresponding to the target frame image, wherein the depth value set records the position information and recorded depth values ​​of the target points in the target area, and the target points correspond to the each pixel point in the feature area;

[0168] The first determination module 703 is configured to determine a target depth map corresponding to the target frame image based on the color value of each pixel in the target frame image and the depth value set.

[0169] Optionally, the depth value acquisition module is further configured to:

[0170] Determine key points in the historical frame image;

[0171] An expansion area including the key point and its adjacent pixels is determined based on a predefined expansion rule, and the expansion area is determined as the feature area.

[0172] Optionally, the depth value acquisition module is further configured to:

[0173] Searching for a target area in the target frame image that matches the feature area;

[0174] Determine the pixel point corresponding to each target point in the target area in the feature area;

[0175] The depth value of the pixel point corresponding to the target point is determined as the depth value of the target point.

[0176] Optionally, the first determining module 703 is further configured to:

[0177] Determine the estimated depth value of each pixel point according to the color value of each pixel point in the target frame image;

[0178] Determining a target depth value of the target point according to the estimated depth value of the target point and the depth value set;

[0179] A target depth map corresponding to the target frame image is constructed based on the target depth value of the target point and the estimated depth value of the non-target point outside the target area in the target frame image.

[0180] Optionally, a difference between a target depth value of the target point and the recorded depth value is positively correlated with a difference between an estimated depth value of the target point and the recorded depth value.

[0181] Optionally, the first determining module 703 is further configured to:

[0182] The color value of each pixel in the target frame image and the depth value set are input into a pre-trained depth prediction model, and an estimated depth map corresponding to the target frame image output by the depth prediction model is obtained, wherein the estimated depth map contains the estimated depth value of each pixel.

[0183] Optionally, it also includes: a model training module configured to:

[0184] Determine a training sample including a color value of each pixel point in a sample target frame image and a sample depth value set, wherein the sample depth value set records position information and a sample depth value of the sample target point in the sample target frame image, and the sample depth value is a depth value of a pixel point matching the sample target point in a sample history frame image;

[0185] Inputting the training sample into the model to be trained to obtain a sample predicted depth map corresponding to the sample target frame image output by the model to be trained;

[0186] The model to be trained is trained according to the sample standard depth map and the sample predicted depth map to obtain the depth prediction model, wherein the sample standard depth map includes the sample actual depth map of the sample target frame image, or a fused depth map corresponding to the sample target frame image, and the fused depth map is obtained by nonlinear fusion of the sample actual depth map and the sample historical depth map corresponding to the sample historical frame image.

[0187] Optionally, the sample standard depth map is the fused depth map, and the difference between the fused depth value corresponding to the sample target point in the fused depth map and the sample historical depth value of the sample target point is positively correlated with the difference between the fused depth value and the actual depth value of the sample target point.

[0188] Optionally, random noise is added to the sample historical depth values ​​recorded in the sample depth value set.

[0189] Optionally, also include:

[0190] The guided filtering module 704 is configured to perform guided filtering on the feature depth map of the historical frame image according to the color value of each pixel in the historical frame image, so as to filter out noise and / or texture of the feature depth map.

[0191] Optionally, the target frame image is a first frame image of the target video, and the device further includes:

[0192] The depth value estimation module 705 is configured to obtain the estimated depth values ​​of all the pixels in the target frame image according to the color values ​​of all the pixels in the target frame image;

[0193] The second determination module 706 is configured to determine the depth map including the estimated depth values ​​of all the pixels as the target depth map.

[0194] The embodiment of the present disclosure further provides an electronic device, including:

[0195] processor;

[0196] a memory for storing instructions executable by the processor;

[0197] The processor is configured to execute the instructions to implement the method for determining image depth information as described in any of the above embodiments.

[0198] An embodiment of the present disclosure further provides a storage medium. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the method for determining image depth information described in any of the above embodiments.

[0199] An embodiment of the present disclosure further provides a computer program product, which is configured to execute the method for determining image depth information described in any of the above embodiments.

[0200] Figure 8 800 is a schematic block diagram of an electronic device according to an embodiment of the present disclosure. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0201] Reference Figure 8 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 818 .

[0202] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method for determining image depth information. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0203] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0204] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0205] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0206] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 818. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0207] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0208] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0209] The communication component 818 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 818 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 818 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0210] In one embodiment of the present disclosure, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned method for determining image depth information.

[0211] In an embodiment of the present disclosure, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by a processor 820 of an electronic device 800 to complete the above-mentioned method for determining image depth information. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0212] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0213] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

[0214] It should be noted that, in the present disclosure, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprises one..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0215] The method and device provided in the embodiments of the present disclosure are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present disclosure. The description of the above embodiments is only used to help understand the method of the present disclosure and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present disclosure, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present disclosure.

Claims

1. A method for determining image depth information, It is characterized in that include: Determine a feature region including the key point and its adjacent pixels according to the key point in the historical frame image, and obtain a depth value of each pixel in the feature region, wherein the historical frame image is located before the target frame image in the target video, and the target frame image includes a target region matching the feature region; Mapping the depth value of each pixel point to the target area according to the matching relationship between the feature area and the target area, generating a depth value set corresponding to the target frame image, wherein the depth value set records the position information and recorded depth values ​​of the target points in the target area, and the target points correspond to the each pixel point in the feature area; Determining a target depth map corresponding to the target frame image based on the color value of each pixel in the target frame image and the depth value set, including: determining an estimated depth value of each pixel in the target frame image according to the color value of each pixel; determining a target depth value of the target point according to the estimated depth value of the target point and the depth value set; constructing a target depth map corresponding to the target frame image based on the target depth value of the target point and the estimated depth values ​​of non-target points in the target frame image outside the target area; Among them, determining the estimated depth value of each pixel point according to the color value of each pixel point in the target frame image includes: inputting the color value of each pixel point in the target frame image and the depth value set into a pre-trained depth prediction model, and obtaining an estimated depth map corresponding to the target frame image output by the depth prediction model, wherein the estimated depth map contains the estimated depth value of each pixel point.

2. The method according to claim 1, It is characterized in that The determining of a feature region including the key point and its adjacent pixels according to the key point in the historical frame image includes: Determine key points in the historical frame image; An expansion area including the key point and its adjacent pixels is determined based on a predefined expansion rule, and the expansion area is determined as the feature area.

3. The method according to claim 2, It is characterized in that The depth value of the target point is determined by: Searching for a target area in the target frame image that matches the feature area; Determine the pixel point corresponding to each target point in the target area in the feature area; The depth value of the pixel point corresponding to the target point is determined as the depth value of the target point.

4. The method according to claim 1, It is characterized in that The difference between the target depth value of the target point and the recorded depth value is positively correlated with the difference between the estimated depth value of the target point and the recorded depth value.

5. The method according to claim 1, It is characterized in that The training method of the depth prediction model comprises: Determine a training sample including a color value of each pixel point in a sample target frame image and a sample depth value set, wherein the sample depth value set records position information of the sample target point in the sample target frame image and a sample historical depth value, and the sample historical depth value is a depth value of a pixel point in the sample historical frame image that matches the sample target point; Inputting the training sample into the model to be trained to obtain a sample predicted depth map corresponding to the sample target frame image output by the model to be trained; The model to be trained is trained according to the sample standard depth map and the sample predicted depth map to obtain the depth prediction model, wherein the sample standard depth map includes the sample actual depth map of the sample target frame image, or a fused depth map corresponding to the sample target frame image, and the fused depth map is obtained by nonlinear fusion of the sample actual depth map and the sample historical depth map corresponding to the sample historical frame image.

6. The method according to claim 5, It is characterized in that The sample standard depth map is the fused depth map, and the difference between the fused depth value corresponding to the sample target point in the fused depth map and the sample historical depth value of the sample target point is positively correlated with the difference between the fused depth value and the actual depth value of the sample target point.

7. The method according to claim 5, It is characterized in that The sample historical depth values ​​recorded in the sample depth value set are added with random noise.

8. The method according to claim 1, It is characterized in that Also includes: According to the color value of each pixel in the historical frame image, a guided filtering process is performed on the feature depth map of the historical frame image to filter out noise and / or texture of the feature depth map.

9. The method according to claim 1, It is characterized in that The target frame image includes a first frame image of the target video, and the method further includes: Obtaining estimated depth values ​​of all pixels in the target frame image according to the color values ​​of all pixels in the target frame image; A depth map including the estimated depth values ​​of all the pixels is determined as the target depth map.

10. A device for determining image depth information, It is characterized in that include: a depth value acquisition module, configured to determine a feature area including the key point and its adjacent pixels according to the key point in the historical frame image, and to acquire the depth value of each pixel in the feature area, wherein the historical frame image is located before the target frame image in the target video, and the target frame image includes a target area matching the feature area; A set generation module is configured to map the depth value of each pixel point to the target area according to the matching relationship between the feature area and the target area, and generate a depth value set corresponding to the target frame image, wherein the depth value set records the position information and recorded depth values ​​of the target points in the target area, and the target points correspond to the each pixel point in the feature area; The first determination module is configured to determine a target depth map corresponding to the target frame image based on the color value of each pixel in the target frame image and the depth value set, and is specifically configured to: determine an estimated depth value of each pixel according to the color value of each pixel in the target frame image; determine a target depth value of the target point according to the estimated depth value of the target point and the depth value set; Constructing a target depth map corresponding to the target frame image based on the target depth value of the target point and the estimated depth value of the non-target point outside the target area in the target frame image; Among them, the first determination module is also configured to: input the color value of each pixel in the target frame image and the depth value set into a pre-trained depth prediction model, and obtain an estimated depth map corresponding to the target frame image output by the depth prediction model, wherein the estimated depth map contains the estimated depth value of each pixel.

11. The device according to claim 10, It is characterized in that The depth value acquisition module is further configured to: Determine key points in the historical frame image; An expansion area including the key point and its adjacent pixels is determined based on a predefined expansion rule, and the expansion area is determined as the feature area.

12. The device according to claim 11, It is characterized in that The depth value acquisition module is further configured to: Searching for a target area in the target frame image that matches the feature area; Determine the pixel point corresponding to each target point in the target area in the feature area; The depth value of the pixel point corresponding to the target point is determined as the depth value of the target point.

13. The device according to claim 12, It is characterized in that The difference between the target depth value of the target point and the recorded depth value is positively correlated with the difference between the estimated depth value of the target point and the recorded depth value.

14. The device according to claim 10, It is characterized in that Also includes: The model training module is configured as follows: Determine a training sample including a color value of each pixel point in a sample target frame image and a sample depth value set, wherein the sample depth value set records position information and a sample depth value of the sample target point in the sample target frame image, and the sample depth value is a depth value of a pixel point matching the sample target point in a sample history frame image; Inputting the training sample into the model to be trained to obtain a sample predicted depth map corresponding to the sample target frame image output by the model to be trained; The model to be trained is trained according to the sample standard depth map and the sample predicted depth map to obtain the depth prediction model, wherein the sample standard depth map includes the sample actual depth map of the sample target frame image, or a fused depth map corresponding to the sample target frame image, and the fused depth map is obtained by nonlinear fusion of the sample actual depth map and the sample historical depth map corresponding to the sample historical frame image.

15. The device according to claim 14, It is characterized in that The sample standard depth map is the fused depth map, and the difference between the fused depth value corresponding to the sample target point in the fused depth map and the sample historical depth value of the sample target point is positively correlated with the difference between the fused depth value and the actual depth value of the sample target point.

16. The device according to claim 14, It is characterized in that The sample historical depth values ​​recorded in the sample depth value set are added with random noise.

17. The device according to claim 10, It is characterized in that Also includes: The guided filtering module is configured to perform guided filtering on the feature depth map of the historical frame image according to the color value of each pixel in the historical frame image, so as to filter out noise and / or texture of the feature depth map.

18. The device according to claim 10, It is characterized in that The target frame image is the first frame image of the target video, and the device further includes: A depth value estimation module is configured to obtain estimated depth values ​​of all pixels in the target frame image according to the color values ​​of all pixels in the target frame image; The second determination module is configured to determine the depth map including the estimated depth values ​​of all the pixels as the target depth map.

19. An electronic device, It is characterized in that include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method for determining image depth information as described in any one of claims 1 to 9.

20. A computer-readable storage medium, It is characterized in that When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method for determining image depth information as claimed in any one of claims 1 to 9.

Citation Information

Patent Citations

  • Stereo video depth image manufacturing method based on area communication image and apparatus thereof

    CN102316352A

  • Monocular vision mapping and positioning method and device, storage medium and mobile equipment

    CN110310304A