Monocular depth estimation method and device, electronic equipment and storage medium
By using a scanning light field camera to acquire multi-view images and perform iterative algorithm processing in monocular depth estimation, combining aberration estimation and image reconstruction, the problem of insufficient accuracy and stability of traditional monocular depth estimation is solved, and more accurate Depth estimation.
Patent Information
- Application Number
- CN202411048471.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-11-26
AI Technical Summary
Traditional monocular depth estimation technology is difficult to reach the depth estimation accuracy of the optical hardware limit in actual scenarios, mainly due to incomplete modeling of point diffusion function and insufficient utilization of depth information of fluctuating optical modeling, resulting in poor accuracy and stability.
Multi-view images are acquired by scanning the light field camera, aberration estimation and image reconstruction are used to use iterative algorithms, and convolution error calculations are performed in combination with the viewing angle image to accurately determine the absolute depth.
The accuracy and robustness of monocular depth estimation are improved, and the problem of insufficient accuracy and stability in traditional methods is overcome.
Smart Images

Figure CN119027477A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer vision technology, and in particular to a monocular depth estimation method, device, electronic device and storage medium. Background Art
[0002] Depth estimation plays an important role in many fields such as autonomous driving, augmented reality, and industrial inspection. Compared with expensive and cumbersome active depth sensors and stereo vision systems, monocular depth estimation technology is a lower-cost and more compact solution. However, traditional monocular depth estimation is limited by the aberrations that may exist in the actual scene or imaging system, and it is difficult to achieve the depth estimation accuracy of the optical hardware limit, which limits the application of monocular depth estimation in many actual scenes. However, the relevant technology still has certain defects in depth estimation, which can be divided into two types: the first is that the imaging system is not fully modeled in the point spread function modeling, and the point spread function of the object away from the focal plane is represented by the defocus aberration in the Zernike polynomial, and the absolute distance of the object is not modeled. The second is based on traditional geometric optics, using parallax estimation to perform rough depth estimation, which does not fully utilize the depth information in the point spread function of wave optics modeling, and because the baseline of the scanning light field camera is small, the accuracy and stability of depth estimation are poor. Therefore, how to improve the accuracy of monocular depth estimation has become a technical problem that cannot be underestimated. Summary of the invention
[0003] In view of this, the purpose of the present application is to provide a monocular depth estimation method, device, electronic device and storage medium, which can use a scanning light field camera to obtain multi-view images, extract the features of each view, use an iterative algorithm to obtain the results of aberration estimation and image reconstruction, and then give an estimate of the absolute depth based on the convolution error of the aberration estimation, image reconstruction results and view images, so as to accurately determine the absolute depth estimation and improve the robustness.
[0004] The present invention provides a monocular depth estimation method, which comprises:
[0005] Based on the multi-angle data collection of the target object by the light field camera, multiple target perspective images are determined;
[0006] Performing iterative deconvolution reconstruction processing and iterative aberration estimation processing on the multiple target perspective images to determine high-resolution images and target aberration estimates corresponding to the multiple target perspective images;
[0007] Based on the target disparity estimation, the high-resolution image and a convolution error calculation between the multiple target perspective images, a target depth estimation corresponding to the multiple target perspective images is determined.
[0008] In a possible implementation manner, performing iterative deconvolution reconstruction processing and iterative aberration estimation processing on the multiple target perspective images to determine the high-resolution images and target aberration estimation corresponding to the multiple target perspective images includes:
[0009] Deconvolutionally reconstructing the plurality of target perspective images based on a first point spread function corresponding to the initial aberration estimation to determine an initial high-resolution image;
[0010] Determine the gradient of the aberration based on the relative translation between each viewing angle image in the initial high-resolution image and the corresponding target viewing angle image, and integrate the gradient of the aberration to obtain an aberration estimate;
[0011] After obtaining the aberration estimation and updating the first point spread function, the deconvolution reconstruction and the aberration estimation processing are iterated until the aberration estimation converges, and the target aberration estimation and the high-resolution image are determined; wherein the point spread function is generated according to the optical parameters of the light field camera, the focus position, and the depth of the object point source.
[0012] In a possible implementation manner, the step of performing convolution error calculation based on the target disparity estimation, the high-resolution image, and the plurality of target perspective images to determine the target depth estimation corresponding to the plurality of target perspective images includes:
[0013] Determining initial depth estimates corresponding to a plurality of the target perspective images based on the target disparity estimate;
[0014] A plurality of second point spread functions are generated within a preset range of the initial depth estimation, and convolution error calculation is performed based on the plurality of second point spread functions, the high-resolution image, and the plurality of target perspective images, and the depth corresponding to the second point spread function with the minimum error value is determined as the target depth estimation.
[0015] In a possible implementation manner, determining the initial depth estimates corresponding to the plurality of target perspective images based on the target disparity estimation includes:
[0016] The defocus term in the target aberration estimation is converted into the depth of an object-side point source in a first point spread function corresponding to the target aberration estimation based on the optical parameters and the focus position of the light field camera, and the depth of the object-side point source in the first point spread function is used as the initial depth estimation.
[0017] In a possible implementation manner, a plurality of second point spread functions are generated within a preset range of the initial depth estimation, a convolution error is calculated based on the plurality of second point spread functions, the high-resolution image, and the plurality of target perspective images, and a depth corresponding to the second point spread function with a minimum error value is determined as the target depth estimation, including:
[0018] Generate a preset number of second point spread functions within a preset range of the initial depth estimation based on the target disparity estimation with defocus terms removed;
[0019] Performing convolution calculation on each of the second point spread functions and the high-resolution image to determine a plurality of low-resolution images;
[0020] An error value between each of the low-resolution images and the plurality of target perspective images is determined, and a depth of an object-side point source in a second point spread function corresponding to the low-resolution image with the smallest error value is determined as the target depth estimate.
[0021] In a possible implementation manner, the multi-angle data collection of the target object based on the light field camera to determine multiple target perspective images includes:
[0022] Based on the light field camera, multi-angle data collection is performed on the target object to determine multiple images;
[0023] The pixel information of each image is arranged, and a plurality of the target viewing angle images are determined from the arranged plurality of images.
[0024] The present application also provides a monocular depth estimation device, the monocular depth estimation device comprising:
[0025] A scanning module is used to collect multi-angle data of a target object based on a light field camera to determine multiple target perspective images;
[0026] A reconstruction module, used for performing iterative deconvolution reconstruction processing and iterative aberration estimation processing on the plurality of target perspective images, and determining high-resolution images and target aberration estimation corresponding to the plurality of target perspective images;
[0027] An estimation module is used to perform convolution error calculation between the target disparity estimation, the high-resolution image and the multiple target perspective images to determine the target depth estimation corresponding to the multiple target perspective images.
[0028] In a possible implementation manner, when the reconstruction module is used to perform iterative deconvolution reconstruction processing and iterative aberration estimation processing on the multiple target perspective images to determine the high-resolution images and target aberration estimation corresponding to the multiple target perspective images, the reconstruction module is specifically used to:
[0029] Determining a gradient of aberration based on relative translation between a plurality of the target perspective images, integrating the gradient of the aberration to determine an initial aberration estimate;
[0030] Deconvolutionally reconstructing the plurality of target perspective images based on a first point spread function corresponding to the initial aberration estimate to determine an initial high-resolution image;
[0031] Based on the initial high-resolution image and the multiple target perspective images, aberration estimation processing is performed and the first point spread function is updated, and the deconvolution reconstruction and the aberration estimation processing are iterated until the aberration estimation converges, so as to determine the target aberration estimation and the high-resolution image; wherein the point spread function is generated according to the optical parameters of the light field camera, the focus position, and the depth of the object point source.
[0032] An embodiment of the present application also provides an electronic device, including: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the monocular depth estimation method as described above are performed.
[0033] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the monocular depth estimation method as described above are executed.
[0034] The embodiment of the present application provides a monocular depth estimation method, device, electronic device and storage medium, the monocular depth estimation method comprising: determining multiple target perspective images based on multi-angle data collection of the target object by a light field camera; performing iterative deconvolution reconstruction processing and iterative disparity estimation processing on the multiple target perspective images to determine the high-resolution images and target disparity estimation corresponding to the multiple target perspective images; wherein the multiple perspective images existing in the high-resolution image are consistent with the perspective in the target perspective image; performing convolution error calculation based on the target disparity estimation, the high-resolution image and the multiple target perspective images to determine the target depth estimation corresponding to the multiple target perspective images. By using a scanning light field camera to obtain multi-perspective images, extracting the features of each perspective, using an iterative algorithm to obtain the results of disparity estimation and image reconstruction, and then giving an absolute depth estimation based on the convolution error combined with the results of disparity estimation, image reconstruction and perspective images, the absolute depth estimation can be accurately determined, thereby improving robustness.
[0035] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0037] Figure 1 A flowchart of a monocular depth estimation method provided in an embodiment of the present application;
[0038] Figure 2 A flowchart of iterative deconvolution reconstruction processing and iterative aberration estimation processing provided in an embodiment of the present application;
[0039] Figure 3 A schematic diagram of the structure of a monocular depth estimation device provided in an embodiment of the present application;
[0040] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0041] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art, under the guidance of the content of the present application, can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.
[0042] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0043] In order to enable those skilled in the art to use the contents of this application, the following implementation is given in combination with the specific application scenario of "performing monocular depth estimation". For those skilled in the art, the general principles defined here can be applied to other embodiments and application scenarios without departing from the spirit and scope of this application.
[0044] The following methods, devices, electronic devices or computer-readable storage media of the embodiments of the present application can be applied to any scenario requiring monocular depth estimation. The embodiments of the present application are not limited to specific application scenarios. Any scheme using a monocular depth estimation method, device, electronic device and storage medium provided by the embodiments of the present application is within the protection scope of the present application.
[0045] First, the application scenarios to which the present application is applicable are introduced. The present application can be applied in the field of computer vision technology.
[0046] Research has found that depth estimation plays an important role in many fields such as autonomous driving, augmented reality, and industrial inspection. Compared with expensive and cumbersome active depth sensors and stereo vision systems, monocular depth estimation technology is a lower-cost and more compact solution. However, traditional monocular depth estimation is limited by the aberrations that may exist in the actual scene or imaging system, and it is difficult to achieve the depth estimation accuracy of the optical hardware limit, which limits the application of monocular depth estimation in many actual scenes. However, the relevant technology still has certain defects in depth estimation, which can be divided into two types: the first is that the imaging system is not fully modeled in the point spread function modeling, and the point spread function of the object away from the focal plane is represented by the defocus aberration in the Zernike polynomial, and the absolute distance of the object is not modeled. The second is based on traditional geometric optics, using parallax estimation to perform rough depth estimation, which does not fully utilize the depth information in the point spread function of wave optics modeling, and because the baseline of the scanning light field camera is small, the accuracy and stability of the depth estimation are poor. Therefore, how to improve the accuracy of monocular depth estimation has become a technical problem that cannot be underestimated.
[0047] Based on this, an embodiment of the present application provides a monocular depth estimation method, which can use a scanning light field camera to obtain multi-view images and extract the features of each view, use an iterative algorithm to obtain the results of aberration estimation and image reconstruction, and then give an estimate of the absolute depth based on the convolution error of the aberration estimation, image reconstruction results and view images. The absolute depth estimation can be accurately determined, thereby improving robustness.
[0048] See also Figure 1 , Figure 1 This is a flow chart of a monocular depth estimation method provided in an embodiment of the present application. Figure 1 As shown in, the monocular depth estimation method provided by the embodiment of the present application includes:
[0049] S101: Determine multiple target perspective images based on multi-angle data collection of a target object by a light field camera.
[0050] In this step, a light field camera is used to collect data from multiple angles on the target object to obtain multiple target perspective images of the target object.
[0051] In a possible implementation manner, the multi-angle data collection of the target object based on the light field camera to determine multiple target perspective images includes:
[0052] Based on the light field camera, multi-angle data of the target object is collected to determine a plurality of images; pixel information of each image is arranged to determine a plurality of target perspective images from the arranged plurality of images.
[0053] Here, the light field camera collects multi-angle data of the target object to obtain multiple images, and the pixel information of each image is arranged to obtain multiple target perspective images.
[0054] S102: Perform iterative deconvolution reconstruction processing and iterative disparity estimation processing on the multiple target perspective images to determine the high-resolution images and target disparity estimation corresponding to the multiple target perspective images; wherein the multiple perspective images existing in the high-resolution image are consistent with the perspective in the target perspective image.
[0055] In this step, an iterative deconvolution reconstruction process and an iterative aberration estimation process are performed on the multiple target view images to determine high-resolution images and target aberration estimates corresponding to the multiple target view images.
[0056] The multiple perspective images in the high-resolution image are consistent with the perspective in the target perspective image.
[0057] Here, deconvolution reconstruction and aberration estimation are performed iteratively. In each iteration, the deconvolution reconstruction is first completed using the current point spread function, and then the aberration is estimated and the point spread function is updated based on the deconvolution reconstruction result. The iteration is performed until the aberration estimation result converges.
[0058] For further information, see Figure 2 , Figure 2 The flowchart of the iterative deconvolution reconstruction process and the iterative aberration estimation process provided in the embodiment of the present application is as follows: Figure 2 As shown:
[0059] S201: Deconvolutionally reconstructing the plurality of target view images based on a first point spread function corresponding to an initial aberration estimate to determine an initial high-resolution image.
[0060] Here, deconvolution reconstruction is performed on multiple target view images according to the first point spread function corresponding to the initial aberration estimate to determine the initial high-resolution image, wherein the gradient of the aberration between the surrounding view images and the intermediate view image is integrated to determine the initial aberration estimate.
[0061] S202: Determine a gradient of the aberration based on a relative translation between each viewing angle image in the initial high-resolution image and the corresponding target viewing angle image, and integrate the gradient of the aberration to obtain an aberration estimate.
[0062] Here, the gradient of the aberration is determined according to the relative translation between each viewing angle image in the initial high-resolution image and the corresponding target viewing angle image, and the gradient of the aberration is integrated to obtain the aberration estimation.
[0063] Among them, each aberration estimate corresponds to a point spread function. For an optical system, the light field distribution of the output image when the input object is a point light source is called a point spread function or a point spread function. Here, the light field point spread function model based on wave optics can generate the point spread function of the light field according to the optical parameters of the camera, the focus position and the depth of the object point source.
[0064] Among them, deconvolution reconstruction uses deconvolution algorithm and estimated aberration to extract different frequency components contained in multi-view images to restore high-resolution clear images.
[0065] S203: updating the first point spread function, iterating the deconvolution reconstruction and the aberration estimation process until the aberration estimation converges, and determining the target aberration estimation and the high-resolution image.
[0066] Here, the first point spread function is updated, and the above steps are repeated to iteratively perform deconvolution reconstruction and aberration estimation processing until the obtained aberration estimation converges, the current aberration estimation is determined as the target aberration estimation, and the currently reconstructed image is determined as the high-resolution image.
[0067] The point spread function is generated according to the optical parameters of the light field camera, the focus position and the depth of the object point source.
[0068] Specifically, the disparity is directly initialized to zero, and then iteration is started to obtain a high-resolution image based on the disparity, and then the disparity that has not been estimated is estimated based on the high-resolution image, the current disparity estimate and the multi-view image (this step is achieved by calculating the relative translation of each view image in the low-resolution multi-view image (synthesized by the point spread function generated by the high-resolution image and the current disparity) and the corresponding view of the collected multi-view image, determining the gradient of the unestimated disparity, and integrating it to obtain the unestimated disparity), and adding it to the current disparity estimate to obtain the target disparity estimate and the high-resolution image.
[0069] S103: performing convolution error calculation based on the target disparity estimation, the high-resolution image, and the plurality of target perspective images to determine target depth estimation corresponding to the plurality of target perspective images.
[0070] In this step, a convolution error calculation is performed based on the target disparity estimation, the high-resolution image, and the multiple target perspective images to determine the target depth estimation corresponding to the multiple target perspective images.
[0071] In a possible implementation manner, the step of performing convolution error calculation based on the target disparity estimation, the high-resolution image, and the plurality of target perspective images to determine the target depth estimation corresponding to the plurality of target perspective images includes:
[0072] a: Based on the target disparity estimation, determine initial depth estimates corresponding to a plurality of the target perspective images.
[0073] Here, initial depth estimates corresponding to a plurality of target perspective images are determined based on the target disparity estimation.
[0074] In a possible implementation manner, determining the initial depth estimates corresponding to the plurality of target perspective images based on the target disparity estimation includes:
[0075] The defocus term in the target aberration estimation is converted into the depth of an object-side point source in a first point spread function corresponding to the target aberration estimation based on the optical parameters and the focus position of the light field camera, and the depth of the object-side point source in the first point spread function is used as the initial depth estimation.
[0076] Here, the defocus term in the target aberration estimation is converted into the depth of the object point source in the first point spread function corresponding to the target aberration estimation according to the optical parameters and the focus position of the light field camera, and the depth of the object point source in the first point spread function is used as the initial depth estimation.
[0077] b: Generate multiple second point spread functions within a preset range of the initial depth estimation, perform convolution error calculation based on the multiple second point spread functions, the high-resolution image and the multiple target perspective images, and determine the depth corresponding to the second point spread function with the minimum error value as the target depth estimation.
[0078] Here, multiple second point spread functions are generated in equal proportion within the preset range of the initial depth estimation, convolution error calculation is performed based on the multiple second point spread functions, the high-resolution image and the multiple target perspective images, and the depth corresponding to the second point spread function with the minimum error value is determined as the target depth estimation.
[0079] In a possible implementation manner, a plurality of second point spread functions are generated within a preset range of the initial depth estimation, a convolution error is calculated based on the plurality of second point spread functions, the high-resolution image, and the plurality of target perspective images, and a depth corresponding to the second point spread function with a minimum error value is determined as the target depth estimation, including:
[0080] (1): Generate a preset number of second point spread functions within a preset range of the initial depth estimation based on the target disparity estimation with defocus terms eliminated.
[0081] Here, a preset number of second point spread functions are generated according to the target disparity estimation with defocus terms removed within a preset range of the initial depth estimation.
[0082] (2): Perform convolution calculation on each of the second point spread functions and the high-resolution image to determine a plurality of low-resolution images.
[0083] Here, each second point spread function is convolved with the high-resolution image to determine a plurality of low-resolution images.
[0084] The multiple perspective images in the low-resolution image are consistent with the perspective in the target perspective image.
[0085] (3): Determine the error value between each of the low-resolution images and the multiple target perspective images, and determine the depth of the object point source in the second point spread function corresponding to the low-resolution image with the smallest error value as the target depth estimate.
[0086] Here, the error value between each low-resolution image and multiple target view images is determined, and the depth of the object point source in the second point spread function corresponding to the low-resolution image with the minimum error value is determined as the target depth estimation.
[0087] Here, according to the optical parameters and focusing position of the camera, an optimization algorithm is used to convert the defocus term in the estimated aberration into the depth of the object point source in the light field point spread function model based on wave optics, which is used as the preliminary depth estimation result. A three-dimensional point spread function is generated near the preliminary depth estimation result based on the estimated aberration after eliminating the defocus term. A low-resolution blurred image is obtained by convolving it with the reconstructed high-resolution clear image. The error between the low-resolution blurred image and the actual captured image is calculated. The depth corresponding to the point spread function with the smallest error is the final depth estimation result.
[0088] In a specific embodiment, the data collected by the light field camera is scanned to obtain a multi-view image. According to the multi-view image, deconvolution reconstruction and aberration estimation are iteratively performed. In each iteration, the deconvolution reconstruction is first completed using the current point spread function, and then the aberration is estimated and the point spread function is updated according to the deconvolution reconstruction result. The aberration estimation result is iterated until the aberration estimation result converges to obtain a high-resolution image and a target aberration estimation. A preliminary depth estimation result is obtained according to the defocus term in the target estimated aberration. A second point spread function generated according to the estimated aberration with the defocus term removed near the preliminary depth estimation result is convolved with the reconstructed high-resolution image and the convolution error is calculated with the actual image. The distance corresponding to the second point spread function with the smallest convolution error is the final depth estimation result.
[0089] The embodiment of the present application provides a monocular depth estimation method, which includes: determining multiple target perspective images based on multi-angle data collection of the target object by a light field camera; performing iterative deconvolution reconstruction processing and iterative aberration estimation processing on the multiple target perspective images to determine the high-resolution images and target aberration estimation corresponding to the multiple target perspective images; wherein the multiple perspective images existing in the high-resolution image are consistent with the perspective in the target perspective image; and performing convolution error calculation between the target aberration estimation, the high-resolution image and the multiple target perspective images to determine the target depth estimation corresponding to the multiple target perspective images. By using a scanning light field camera to obtain multi-perspective images, extracting the features of each perspective, using an iterative algorithm to obtain the results of aberration estimation and image reconstruction, and then giving an absolute depth estimation based on the convolution error combined with the aberration estimation, the image reconstruction results and the perspective image, the absolute depth estimation can be accurately determined, thereby improving robustness.
[0090] See also Figure 3 , Figure 3 This is a schematic diagram of the structure of a monocular depth estimation device provided in an embodiment of the present application. Figure 3 As shown in , the monocular depth estimation device 300 includes:
[0091] A scanning module 310 is used to collect multi-angle data of a target object based on a light field camera to determine multiple target perspective images;
[0092] A reconstruction module 320 is used to perform iterative deconvolution reconstruction processing and iterative disparity estimation processing on the multiple target perspective images to determine high-resolution images and target disparity estimates corresponding to the multiple target perspective images;
[0093] The estimation module 330 is used to perform convolution error calculation between the target disparity estimation, the high-resolution image and the multiple target perspective images to determine the target depth estimation corresponding to the multiple target perspective images.
[0094] Further, when the reconstruction module 320 is used to perform iterative deconvolution reconstruction processing and iterative aberration estimation processing on the multiple target perspective images to determine the high-resolution images and target aberration estimation corresponding to the multiple target perspective images, the reconstruction module 320 is specifically used to:
[0095] Deconvolution reconstruction is performed on the plurality of target view images using a first point spread function corresponding to the initial aberration estimation to determine an initial high-resolution image;
[0096] Determine the gradient of the aberration based on the relative translation between each viewing angle image in the initial high-resolution image and the corresponding target viewing angle image, and integrate the gradient of the aberration to obtain an aberration estimate;
[0097] After obtaining the disparity estimation and updating the first point spread function, the deconvolution reconstruction and the disparity estimation processing are iterated until the disparity estimation converges, and the target disparity estimation and the high-resolution image are determined. Further, when the estimation module 330 is used to perform convolution error calculation based on the target disparity estimation, the high-resolution image and the multiple target perspective images to determine the target depth estimation corresponding to the multiple target perspective images, the estimation module 330 is specifically used to:
[0098] Determining initial depth estimates corresponding to a plurality of the target perspective images based on the target disparity estimate;
[0099] A plurality of second point spread functions are generated within a preset range of the initial depth estimation, and convolution error calculation is performed based on the plurality of second point spread functions, the high-resolution image, and the plurality of target perspective images, and the depth corresponding to the second point spread function with the minimum error value is determined as the target depth estimation.
[0100] Further, when the estimation module 330 is used to determine the initial depth estimation corresponding to the plurality of target perspective images based on the target disparity estimation, the estimation module 330 is specifically used to:
[0101] The defocus term in the target aberration estimation is converted into the depth of an object-side point source in a first point spread function corresponding to the target aberration estimation based on the optical parameters and the focus position of the light field camera, and the depth of the object-side point source in the first point spread function is used as the initial depth estimation.
[0102] Further, the estimation module 330 generates a plurality of second point spread functions within a preset range for the initial depth estimation, performs convolution error calculation based on the plurality of second point spread functions, the high-resolution image, and the plurality of target perspective images, and determines the depth corresponding to the second point spread function with the minimum error value as the target depth estimation. The estimation module 330 is specifically used to:
[0103] Generate a preset number of second point spread functions within a preset range of the initial depth estimation based on the target disparity estimation with defocus terms removed;
[0104] Performing convolution calculation on each of the second point spread functions and the high-resolution image to determine a plurality of low-resolution images;
[0105] An error value between each of the low-resolution images and the plurality of target perspective images is determined, and a depth of an object-side point source in a second point spread function corresponding to the low-resolution image with the smallest error value is determined as the target depth estimate.
[0106] Furthermore, when the scanning module 310 is used for the multi-angle data collection of the target object based on the light field camera to determine multiple target perspective images, the scanning module 310 is specifically used for:
[0107] Based on the light field camera, multi-angle data collection is performed on the target object to determine multiple images;
[0108] The pixel information of each image is arranged, and a plurality of the target viewing angle images are determined from the arranged plurality of images.
[0109] The embodiment of the present application provides a monocular depth estimation device, which includes: a scanning module for determining multiple target view images based on multi-angle data collection of a target object by a light field camera; a reconstruction module for performing iterative deconvolution reconstruction processing and iterative aberration estimation processing on multiple target view images to determine high-resolution images and target aberration estimation corresponding to multiple target view images; an estimation module for performing convolution error calculation between the target aberration estimation, the high-resolution image and multiple target view images to determine target depth estimation corresponding to multiple target view images. By using a scanning light field camera to obtain multi-view images and extracting features of each view, using an iterative algorithm to obtain aberration estimation and image reconstruction results, and then giving an absolute depth estimate based on the convolution error combined with the aberration estimation, image reconstruction results and view images, the absolute depth estimate can be accurately determined, thereby improving robustness.
[0110] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown in , the electronic device 400 includes a processor 410 , a memory 420 and a bus 430 .
[0111] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, the above-mentioned Figure 1 as well as Figure 2 The steps of the monocular depth estimation method in the method embodiment shown, the specific implementation method can be found in the method embodiment, and will not be repeated here.
[0112] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 as well as Figure 2 The steps of the monocular depth estimation method in the method embodiment shown, the specific implementation method can be found in the method embodiment, and will not be repeated here.
[0113] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0114] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.
[0115] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0116] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0117] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application can essentially be embodied in the form of a software product, or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0118] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A monocular depth estimation method, It is characterized in that The monocular depth estimation method comprises: Based on the multi-angle data collection of the target object by the light field camera, multiple target perspective images are determined; Performing iterative deconvolution reconstruction processing and iterative aberration estimation processing on the multiple target perspective images to determine high-resolution images and target aberration estimates corresponding to the multiple target perspective images; Based on the target disparity estimation, the high-resolution image and a convolution error calculation between the multiple target perspective images, a target depth estimation corresponding to the multiple target perspective images is determined.
2. The monocular depth estimation method according to claim 1, It is characterized in that The iterative deconvolution reconstruction process and iterative aberration estimation process are performed on the multiple target perspective images to determine the high-resolution images and target aberration estimation corresponding to the multiple target perspective images, including: Deconvolutionally reconstructing the plurality of target perspective images based on a first point spread function corresponding to the initial aberration estimation to determine an initial high-resolution image; Determine the gradient of the aberration based on the relative translation between each viewing angle image in the initial high-resolution image and the corresponding target viewing angle image, and integrate the gradient of the aberration to obtain an aberration estimate; After obtaining the aberration estimation and updating the first point spread function, the deconvolution reconstruction and the aberration estimation process are iterated until the aberration estimation converges, thereby determining the target aberration estimation and the high-resolution image.
3. The monocular depth estimation method according to claim 1, It is characterized in that The method of performing convolution error calculation based on the target disparity estimation, the high-resolution image, and the plurality of target perspective images to determine target depth estimation corresponding to the plurality of target perspective images includes: Determining initial depth estimates corresponding to a plurality of the target perspective images based on the target disparity estimate; A plurality of second point spread functions are generated within a preset range of the initial depth estimation, and convolution error calculation is performed based on the plurality of second point spread functions, the high-resolution image, and the plurality of target perspective images, and the depth corresponding to the second point spread function with the minimum error value is determined as the target depth estimation.
4. The monocular depth estimation method according to claim 3, It is characterized in that The step of determining initial depth estimates corresponding to a plurality of target perspective images based on the target disparity estimation includes: The defocus term in the target aberration estimation is converted into the depth of an object-side point source in a first point spread function corresponding to the target aberration estimation based on the optical parameters and the focus position of the light field camera, and the depth of the object-side point source in the first point spread function is used as the initial depth estimation.
5. The monocular depth estimation method according to claim 3, It is characterized in that Generating a plurality of second point spread functions within a preset range of the initial depth estimation, performing convolution error calculation based on the plurality of second point spread functions, the high-resolution image, and the plurality of target perspective images, and determining the depth corresponding to the second point spread function with a minimum error value as the target depth estimation, including: Generate a preset number of second point spread functions within a preset range of the initial depth estimation based on the target disparity estimation with defocus terms removed; Performing convolution calculation on each of the second point spread functions and the high-resolution image to determine a plurality of low-resolution images; An error value between each of the low-resolution images and the plurality of target perspective images is determined, and a depth of an object-side point source in a second point spread function corresponding to the low-resolution image with the smallest error value is determined as the target depth estimate.
6. The monocular depth estimation method according to claim 1, It is characterized in that The method of collecting multi-angle data of the target object based on the light field camera to determine multiple target perspective images includes: Based on the light field camera, multi-angle data collection is performed on the target object to determine multiple images; The pixel information of each image is arranged, and a plurality of the target viewing angle images are determined from the arranged plurality of images.
7. A monocular depth estimation device, It is characterized in that The monocular depth estimation device comprises: A scanning module is used to collect multi-angle data of a target object based on a light field camera to determine multiple target perspective images; A reconstruction module, used for performing iterative deconvolution reconstruction processing and iterative aberration estimation processing on the plurality of target perspective images, and determining high-resolution images and target aberration estimation corresponding to the plurality of target perspective images; An estimation module is used to perform convolution error calculation between the target disparity estimation, the high-resolution image and the multiple target perspective images to determine the target depth estimation corresponding to the multiple target perspective images.
8. The monocular depth estimation device according to claim 7, It is characterized in that When the reconstruction module is used to perform iterative deconvolution reconstruction processing and iterative aberration estimation processing on the multiple target perspective images to determine the high-resolution images and target aberration estimation corresponding to the multiple target perspective images, the reconstruction module is specifically used to: Determining a gradient of aberration based on relative translation between a plurality of the target perspective images, integrating the gradient of the aberration to determine an initial aberration estimate; Deconvolutionally reconstructing the plurality of target perspective images based on a first point spread function corresponding to the initial aberration estimate to determine an initial high-resolution image; Based on the initial high-resolution image and the multiple target perspective images, aberration estimation processing is performed and the first point spread function is updated, and the deconvolution reconstruction and the aberration estimation processing are iterated until the aberration estimation converges, so as to determine the target aberration estimation and the high-resolution image; wherein the point spread function is generated according to the optical parameters of the light field camera, the focus position, and the depth of the object point source.
9. An electronic device, It is characterized in that include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the monocular depth estimation method as described in any one of claims 1 to 6.
10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the monocular depth estimation method according to any one of claims 1 to 6 are executed.
