Image depth estimation method and device, electronic device, and storage medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI WINGTECH ELECTRONICS TECH
- Filing Date
- 2023-06-27
- Publication Date
- 2026-07-21
Smart Images
Figure CN116761085B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of imaging technology, and in particular to an image depth estimation method and apparatus, electronic device, and storage medium. Background Technology
[0002] Currently, with the rapid development of dual-camera technology, users can quickly acquire images with depth information using the binoculars on their electronic devices when taking pictures. However, in practice, it has been found that in certain shooting scenarios (such as those with specular reflections, obstructed viewpoints, or weak textures), the accuracy of the image depth determined by the electronic device's binoculars tends to decrease, thus hindering accurate image depth information estimation. Summary of the Invention
[0003] This application discloses an image depth estimation method, apparatus, electronic device, and storage medium, which can improve the accuracy of electronic devices in estimating image depth information.
[0004] The first aspect of this application discloses an image depth estimation method, including:
[0005] In the image region to be estimated, at least one neighboring hole point corresponding to the target pixel is determined; wherein, the target pixel is any pixel in the image region to be estimated, and the initial disparity value corresponding to the target pixel and the at least one neighboring hole point is 0, and the initial disparity value is the disparity value corresponding to the pixel determined when the image is captured by a binocular device.
[0006] Determine the target cavity point from the at least one neighboring cavity point, and determine at least one neighboring estimated point corresponding to the target cavity point;
[0007] Based on the initial disparity value corresponding to the at least one neighboring estimated point, the estimated disparity value corresponding to the target hole point is calculated, and the estimated disparity value is used to determine the image depth of the target hole point in the image region to be estimated.
[0008] A second aspect of this application discloses an image depth estimation apparatus, comprising:
[0009] The first determining unit is used to determine at least one neighboring hole point corresponding to a target pixel point in the image region to be estimated; wherein, the target pixel point is any pixel point in the image region to be estimated, the initial disparity value corresponding to the target pixel point and the at least one neighboring hole point is 0, and the initial disparity value is the disparity value corresponding to the pixel point determined when the image is captured by a binocular device.
[0010] The second determining unit is used to determine the target cavity point from the at least one neighboring cavity point, and to determine at least one neighboring estimated point corresponding to the target cavity point;
[0011] The calculation unit is used to calculate the estimated disparity value corresponding to the target hole point based on the initial disparity value corresponding to the at least one neighboring estimated point, and the estimated disparity value is used to determine the image depth of the target hole point in the image region to be estimated.
[0012] The third aspect of this application discloses an electronic device, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs all or part of the steps in any of the image depth estimation methods disclosed in the first aspect of this application.
[0013] The fourth aspect of this application discloses a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements all or part of the steps in any of the image depth estimation methods disclosed in the first aspect of this application.
[0014] Compared with related technologies, the embodiments of this application have the following beneficial effects:
[0015] In this embodiment, the electronic device can determine at least one neighboring hole point corresponding to a target pixel in the image region to be estimated. The target pixel can be any pixel in the image region to be estimated. The initial disparity value corresponding to both the target pixel and the at least one neighboring hole point is 0. The initial disparity value is the disparity value corresponding to the pixel determined when the image is captured using a binocular device. The electronic device can determine the target hole point from the at least one neighboring hole point and determine at least one neighboring estimated point corresponding to the target hole point. Based on this, the electronic device can calculate the estimated disparity value corresponding to the target hole point based on the initial disparity value corresponding to the at least one neighboring estimated point. This estimated disparity value is used to determine the image depth of the target hole point in the image region to be estimated. Therefore, by implementing this embodiment, the electronic device can estimate the missing disparity data of the hole pixel in the image region to be estimated by using the relatively accurate disparity data of pixels near the region edge, thereby improving the accuracy of the electronic device in estimating image depth information. This not only reduces the computational cost of filling parallax holes and enables rapid and accurate image depth information estimation, but also applies to parallax holes of different sizes, helping to further improve the accuracy and convenience of depth information estimation for electronic devices equipped with binoculars. Attached Figure Description
[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0017] Figure 1A This is a schematic diagram illustrating an application scenario of the image depth estimation method disclosed in the embodiments of this application;
[0018] Figure 1B This is a schematic diagram of the structure of an electronic device that applies an image depth estimation method as disclosed in an embodiment of this application;
[0019] Figure 2 This is a schematic flowchart of an image depth estimation method disclosed in an embodiment of this application;
[0020] Figure 3 This is a schematic diagram of an image region to be estimated in a target image disclosed in an embodiment of this application;
[0021] Figure 4 This is a flowchart illustrating another image depth estimation method disclosed in an embodiment of this application;
[0022] Figure 5A This is a schematic diagram of determining at least one neighboring hole point in the image region to be estimated, as disclosed in an embodiment of this application;
[0023] Figure 5B This is a schematic diagram of determining at least one neighboring estimated point corresponding to a target cavity point as disclosed in an embodiment of this application;
[0024] Figure 6 This is a flowchart illustrating another image depth estimation method disclosed in an embodiment of this application;
[0025] Figure 7 This is a schematic diagram illustrating the process of filling the image region to be estimated as disclosed in an embodiment of this application;
[0026] Figure 8 This is a modular schematic diagram of an image depth estimation device disclosed in an embodiment of this application;
[0027] Figure 9 This is a modular schematic diagram of an electronic device disclosed in an embodiment of this application. Detailed Implementation
[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0029] It should be noted that the terms "comprising" and "having" and any variations thereof in the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or devices.
[0030] This application discloses an image depth estimation method, apparatus, electronic device, and storage medium, which can improve the accuracy of electronic devices in estimating image depth information.
[0031] The following will be described in detail with reference to the accompanying drawings.
[0032] Please see Figure 1A , Figure 1A This is a schematic diagram illustrating an application scenario of the image depth estimation method disclosed in an embodiment of this application. For example... Figure 1A As shown, the electronic device 10 may include a binocular device 11 (including a first camera and a second camera). Figure 1A (Not specifically shown in the image), the electronic device 10 can use the binocular device 11 to capture a target image, and then determine the disparity data corresponding to each pixel in the target image based on the principle of binocular parallax. It is understood that due to factors such as specular reflection, viewing angle occlusion, or the presence of weak texture areas in the captured target, the target image may contain disparity holes (or depth information holes) in which depth information cannot be determined. The electronic device 10 can also be used to estimate the depth information of the image region to be estimated containing disparity holes, so as to determine the disparity data corresponding to each pixel in the image region to be estimated, and obtain a complete depth map corresponding to the target image.
[0033] For example, the electronic device 10 may include various devices or systems equipped with binocular devices 11, such as smartphones, smart wearable devices, tablets, PCs (Personal Computers), and various SoCs (System-on-Chips) with image processing capabilities, etc., which are not specifically limited in this embodiment. It should be noted that... Figure 1AThe electronic device 10 shown is a smartphone. This is merely an example and should not be considered as a limitation on the device type of the electronic device 10 in the embodiments of this application.
[0034] In some embodiments, the electronic device 10 may not have a binocular device 11, but is instead used to receive target images sent by other devices and to estimate depth information of the image region to be estimated in the target image.
[0035] In some embodiments, after the electronic device 10 captures a target image through its binocular device 11, it can transmit the target image to other devices (such as servers) so that the other devices can assist in estimating the depth information of the image region to be estimated in the target image and then obtain the corresponding disparity data.
[0036] Taking the above image depth estimation process performed on electronic device 10 as an example, please refer to... Figure 1B , Figure 1B This is a schematic diagram of the structure of an electronic device 10 that applies the image depth estimation method disclosed in this application. Figure 1B As shown, in addition to the binocular device 11 mentioned above, the electronic device 10 may further include necessary modules such as an image signal processing (ISP) processor 12 and an image memory 13, and may also include other modules such as a display 14.
[0037] The binocular device 11 may include a first camera 11a and a second camera 11b. Typically, when taking pictures using the binocular device 11, the disparity data corresponding to the corresponding pixels of the target in the captured image can be determined based on the different distances between the first camera 11a and the second camera 11b and the target, and thus the image depth of the aforementioned pixels in the target image can be determined.
[0038] In some embodiments, the target image captured by the binocular device 11 can be processed by the ISP processor 12. Exemplarily, the ISP processor 12 can first determine the image region to be estimated in the target image where a disparity hole exists. In this embodiment, the ISP processor 12 can determine at least one neighboring hole point corresponding to a target pixel within the image region to be estimated. The target pixel can be any pixel within the image region to be estimated, and the initial disparity value corresponding to both the target pixel and the at least one neighboring hole point is 0. The initial disparity value is the disparity value corresponding to the pixel determined when the image is captured by the binocular device. The ISP processor 12 can also determine the target hole point from the at least one neighboring hole point and determine at least one neighboring estimated point corresponding to the target hole point. Based on this, according to the initial disparity value corresponding to the at least one neighboring estimated point, an estimated disparity value corresponding to the target hole point can be calculated. This estimated disparity value is used to determine the image depth of the target hole point in the image region to be estimated.
[0039] In other embodiments, the ISP processor 12 may also acquire target images from the image memory 13. For example, the binocular device 11 may cache the target images it captures in the image memory 13, and then the image memory 13 may provide the stored target images to the ISP processor 12 for processing. Exemplarily, the image memory 13 may be part of a memory device (not specifically illustrated), a storage device, or a separate dedicated memory within an electronic device, and may include DMA (Direct Memory Access) features, which are not specifically limited in the embodiments of this application.
[0040] Optionally, the disparity data obtained after depth information estimation can also be sent to the image memory 13 for further processing before display. In some embodiments, the ISP processor 12 can output the target image and corresponding disparity data to the display 14 for user viewing and / or further processing by the graphics engine or GPU (Graphics Processing Unit). In other embodiments, the output of the ISP processor 12 can also be sent to the image memory 13, and the display 14 can read the target image and corresponding disparity data from the image memory 13.
[0041] By implementing the aforementioned image depth estimation method, the electronic device 10 can estimate the missing disparity data of the holed pixels in the image region to be estimated by utilizing the relatively accurate disparity data of pixels near the region edge, thereby improving the accuracy of image depth estimation by the electronic device 10. This not only reduces the computational load required to fill disparity holes and completes the image depth estimation process quickly and accurately, but also applies to disparity holes of different sizes, further enhancing the accuracy and convenience of depth information estimation by the electronic device 10 equipped with the binocular device 11.
[0042] Please see Figure 2 , Figure 2 This is a flowchart illustrating an image depth estimation method disclosed in an embodiment of this application, which can be applied to the aforementioned electronic device. Figure 2 As shown, the image depth estimation method may include the following steps:
[0043] 202. In the image region to be estimated, determine at least one neighboring hole point corresponding to the target pixel; wherein, the target pixel is any pixel in the image region to be estimated, and the initial disparity value corresponding to the target pixel and the above-mentioned at least one neighboring hole point is 0, and the initial disparity value is the disparity value corresponding to the pixel determined when the image is captured by a binocular device.
[0044] In this embodiment of the application, the electronic device may be configured with a binocular device, thereby enabling the capture of a target image through the binocular device. For example, as shown... Figure 3 As shown, in the target image, due to the possible presence of specular reflection areas, viewpoint occlusion areas, weak texture areas, etc., disparity holes (or depth information holes) are likely to occur. That is, because the disparity cannot be determined based on the binocular device, or the initial disparity value determined by the binocular device is 0, the image region to be estimated composed of pixels with depth information cannot be further determined.
[0045] In order to estimate the depth information of each pixel in the image region to be estimated, the electronic device can select any pixel in the image region to be estimated as the target pixel, and then determine the corresponding disparity data (which can be denoted as "estimated disparity value") of the neighboring hole points that meet the specified conditions in the vicinity of the target pixel in subsequent steps. Thus, by determining the estimated disparity value corresponding to each pixel in the image region to be estimated one by one, the process of "filling" the image region to be estimated, that is, the disparity hole, can be gradually completed.
[0046] In some embodiments, the electronic device can determine at least one neighboring hole point corresponding to the target pixel along a specified direction. For example, the electronic device can start from the target pixel and determine pixels located at the edge of the image region to be estimated along specified directions such as directly above, directly below, directly left, directly right, upper left, lower left, upper right, and lower right, or along other specified directions, as neighboring hole points corresponding to the target pixel. It is understood that since pixels at the edge of disparity holes are less likely to be affected by specular reflection areas, viewing angle occlusion areas, weak texture areas, etc., in the target image, the corresponding disparity data is relatively accurate and can be used to more accurately estimate the estimated disparity value corresponding to neighboring hole points.
[0047] In other embodiments, after determining the image region to be estimated, the electronic device can also filter out suitable neighboring hole points by traversing the pixels along the edge of the image region to be estimated. For example, the electronic device can use algorithms such as clustering and downsampling to filter out representative pixels along the edge of the image region to be estimated and determine them as neighboring hole points corresponding to the target pixel for subsequent depth information estimation.
[0048] 204. Determine the target cavity from at least one neighboring cavity, and determine at least one neighboring estimated point corresponding to the target cavity.
[0049] In this embodiment, after determining at least one neighboring hole point corresponding to a target pixel, the electronic device can further select the target hole point and estimate the disparity data corresponding to the target hole point. For example, the electronic device can determine at least one pixel point near the target hole point as a neighboring estimation point, so as to estimate the estimated disparity value corresponding to the target hole point using the relatively accurate disparity data of the pixels at the edge of the disparity hole.
[0050] In some embodiments, the electronic device can start from the target hole point and determine the pixel points that are a certain distance away from the target hole point in the specified directions such as directly above, directly below, directly left, directly right, upper left, lower left, upper right, and lower right, or in other specified directions, as the nearest estimated points corresponding to the target hole point.
[0051] In other embodiments, the electronic device may also traverse pixels at a predetermined step size (e.g., 2 pixels, 4 pixels, etc.) that are a certain distance from the target hole, thereby selecting at least one suitable neighboring estimated point. For example, the electronic device may obtain the initial disparity value corresponding to each traversed pixel and discard pixels whose initial disparity value does not fall within a specified threshold range, thus using the remaining pixels as neighboring estimated points corresponding to the target hole. This helps to reduce interference from disparity holes and other disparity anomalies, improving the accuracy of depth information estimation by the electronic device.
[0052] 206. Based on the initial disparity value corresponding to at least one neighboring estimated point, calculate the estimated disparity value corresponding to the target hole point. This estimated disparity value is used to determine the image depth of the target hole point in the image region to be estimated.
[0053] In this embodiment, based on the initial disparity values corresponding to at least one neighboring hole point near the target hole point, the disparity data corresponding to the target hole point can be estimated. The disparity data may include estimated disparity values, representing the possible image depth at the target hole point when captured by a binocular device. For example, the electronic device can calculate the estimated disparity value corresponding to the target hole point by averaging or taking the median of the initial disparity values corresponding to the at least one neighboring estimated points.
[0054] Based on this, by repeatedly executing steps 202 to 206, depth information can be continuously calculated for the newly determined target hole points, and the estimated disparity values corresponding to each pixel in the image region to be estimated can be gradually determined, thereby completing the overall depth information calculation of the image region to be estimated, so as to adaptively fill disparity holes of different sizes.
[0055] As can be seen, by implementing the image depth estimation method described in the above embodiments, the electronic device can estimate the missing disparity data of the holed pixels in the image region to be estimated by using the relatively accurate disparity data of the pixels near the edge of the region, thereby improving the accuracy of the electronic device in estimating image depth information. This not only reduces the computational load required to fill disparity holes and completes the image depth information estimation process quickly and accurately, but also applies to disparity holes of different sizes, further improving the accuracy and convenience of depth information estimation in electronic devices equipped with binocular vision.
[0056] Please see Figure 4 , Figure 4 This is a flowchart illustrating another image depth estimation method disclosed in an embodiment of this application, which can be applied to the aforementioned electronic device. Figure 4 As shown, the image depth estimation method may include the following steps:
[0057] 402. In the image region to be estimated, with the target pixel as the center, along at least one first target direction, obtain the pixel with an initial disparity value of 0 and the image distance farthest from the target pixel in each first target direction, and use it as the neighboring hole point of the target pixel in each first target direction.
[0058] In this embodiment of the application, in order to determine at least one neighboring hole point corresponding to a target pixel, the electronic device may take the target pixel as the center and select at least one pixel point located in the first target direction and at the edge of the image region to be estimated as the neighboring hole point corresponding to the target pixel in each of the first target directions.
[0059] For example, the aforementioned first target direction may include multiple specified directions such as directly above, directly below, directly to the left, directly to the right, upper left, lower left, upper right, and lower right, and may also include other specified directions. Specifically, for example... Figure 5A As shown, the pixel A0, which has an initial disparity of 0 directly to the right of the target pixel A and is the farthest from the target pixel A (i.e., located directly above the edge of the image region to be estimated), can be considered as the neighboring hole point directly to the right of the target pixel A. Similarly, the neighboring hole points corresponding to the upper right, upper left, upper left, lower left, lower right, and lower right of the target pixel A can be represented as points A1 to A7, respectively.
[0060] 404. Among the above at least one neighboring hole points, the neighboring hole point with the farthest image distance from the target pixel point is determined as the target hole point.
[0061] In this embodiment, to minimize interference from drastic changes in disparity data at the edge of the disparity hole, a relatively large target distance can be used to select at least one neighboring estimation point near the target hole point for estimating its estimated disparity value. Prior to this, a suitable target hole point can be determined based on the image distance between each of the at least one neighboring hole point and the target pixel.
[0062] For example, the electronic device can select a neighboring hole point that is relatively far from the target pixel from the at least one neighboring hole point, such as the neighboring hole point with the farthest image distance from the target pixel, as the target hole point. Then, in subsequent steps, the image distance between the target hole point and the target pixel can be used as the target distance to conveniently determine at least one estimated neighboring point corresponding to the target hole point.
[0063] For specific examples, such as Figure 5A As shown, among the neighboring hole points A0 to A7, A2 has the farthest image distance from the target pixel point A. Thus, the electronic device can determine A2 as the current target hole point and use the image distance between the target hole point A2 and the target pixel point A as the target distance. In subsequent steps, the neighboring estimation points required for estimating depth information are further determined.
[0064] 406. Taking the target hole point as the center, along at least one second target direction, obtain the pixel points whose image distance from the target hole point is equal to the target distance in each second target direction, and use them as the nearest estimated points of the target hole point in each second target direction.
[0065] In this embodiment of the application, in order to determine at least one neighboring estimated point corresponding to the target hole point, the electronic device may take the target hole point as the center and select at least one pixel point located in the second target direction and at a certain distance from the target hole point as the neighboring estimated point corresponding to the target hole point in each second target direction.
[0066] For example, the aforementioned second target direction may include multiple specified directions such as directly above, directly below, directly to the left, directly to the right, upper left, lower left, upper right, and lower right, and may also include other specified directions. Specifically, for example... Figure 5B As shown, pixel B0, located directly to the right of the target hole A2 and at a distance from A2 relative to the target distance (i.e., situated on a circle centered at A2 with the target distance as its radius), can be considered as the nearest estimated point to the right of the target hole A2. Similarly, the nearest estimated points to the upper right, directly above, upper left, directly left, lower left, directly below, and lower right of the target hole A2 can be represented as points B1 to B7, respectively.
[0067] Wherein, the aforementioned target distance can be the image distance between target hole point A2 and target pixel point A, that is, the distance between target hole point A2 and its neighboring estimated point B2 (e.g., Figure 5B The image distance between (as shown).
[0068] 408. Calculate the mean of the initial disparity values corresponding to at least one of the above-mentioned neighboring estimated points, and use it as the estimated disparity value corresponding to the target hole point.
[0069] In this embodiment, the estimated disparity value corresponding to the target hole point can be used to determine the image depth of the target hole point in the image region to be estimated. The electronic device can obtain the estimated disparity value corresponding to the target hole point by calculating the mean of the initial disparity values corresponding to at least one neighboring estimated points.
[0070] For example, taking at least one of the above-mentioned neighbor estimation points including points B0 to B7 as an example (e.g.) Figure 5B As shown in the figure, VB0 to VB7 represent the initial disparity values corresponding to the nearest estimated points B0 to B7, respectively. The above calculation process can be shown in the following formula 1:
[0071] Formula 1:
[0072] VA2=(VB0+VB1+VB2+VB3+VB4+VB5+VB6+VB7) / 8
[0073] Where VA2 represents the target cavity point A2 (e.g., Figure 5B The estimated disparity value VA2 corresponds to the target hole point A2 in the image region to be estimated. This estimated disparity value VA2 can be used to determine the image depth information of the target hole point A2 in the image region to be estimated.
[0074] As can be seen, by implementing the image depth estimation method described in the above embodiments, the electronic device can estimate the missing disparity data of the holed pixels in the image region to be estimated by using the relatively accurate disparity data of pixels near the edge of the region, thereby improving the accuracy of the electronic device in estimating image depth information. At the same time, this estimation method not only reduces the computational load required to fill disparity holes, completing the image depth information estimation process quickly and accurately, but is also applicable to disparity holes of different sizes, further improving the accuracy and convenience of depth information estimation for electronic devices with binocular vision. Furthermore, by selecting neighboring estimation points relatively far from the target hole point for depth information estimation, interference caused by drastic changes in disparity data at the edge of the disparity hole can be avoided as much as possible, further improving the accuracy of depth information estimation by the electronic device.
[0075] Please see Figure 6 , Figure 6 This is a flowchart illustrating another image depth estimation method disclosed in an embodiment of this application, which can be applied to the aforementioned electronic device. Figure 6 As shown, the image depth estimation method may include the following steps:
[0076] 602. In the target image, determine the initial disparity value corresponding to each pixel, and add the pixels with an initial disparity value of 0 to the image region to be estimated.
[0077] In this embodiment, by analyzing and calculating the target image captured by the electronic device using a binocular device, the image region to be estimated containing disparity holes in the target image can be quickly determined. For example, since each pixel in the target image acquired by the binocular device has a uniquely determined initial disparity value (pixels whose disparity cannot be determined based on the binocular device can be considered as having an initial disparity value of 0), the electronic device can, after determining the initial disparity value corresponding to each pixel in the target image, add pixels with an initial disparity value of 0 to the image region to be estimated, thereby finding possible disparity holes in the target image.
[0078] 604. In the image region to be estimated, with the target pixel as the center, along at least one first target direction, obtain the pixel with an initial disparity value of 0 and the image distance from the target pixel in each first target direction, and use it as the neighboring hole point of the target pixel in each first target direction.
[0079] Step 604 is similar to step 402 above, and will not be described again here.
[0080] 606. Among the above at least one neighboring hole points, the neighboring hole point with the farthest image distance from the target pixel point is determined as the target hole point.
[0081] 608. Taking the target hole point as the center, along at least one second target direction, obtain the pixel points whose image distance from the target hole point is equal to the target distance in each second target direction, and use them as the nearest estimated points of the target hole point in each second target direction.
[0082] The target distance mentioned above represents the image distance between the target hole point and the target pixel point. It can be understood that steps 606 and 608 are similar to steps 404 and 406 above, and will not be repeated here.
[0083] 610. Calculate the mean of the initial disparity values corresponding to at least one of the above-mentioned neighboring estimated points, and use it as the estimated disparity value corresponding to the target hole point.
[0084] Step 610 is similar to step 408 above. It should be noted that the electronic device can repeat steps 604 to 610 for the above-mentioned image area to be estimated. That is, after estimating the estimated disparity value corresponding to the target hole point, a new target hole point is re-determined, so as to continue to use the neighboring estimated points near the new target hole point to determine its corresponding estimated disparity value, thereby gradually completing the "filling" of the disparity hole.
[0085] For example, the above filling process can be referred to Figure 7 .like Figure 7As shown, as the filling process proceeds, the image region to be estimated containing parallax holes will gradually shrink (e.g., Figure 7 The process involves filling the image region to be estimated by different lines (representing different areas of the image region to be estimated, which change from first to last) until all the neighboring holes near the target pixel A are filled (i.e., their estimated disparity values are estimated). Finally, by estimating the estimated disparity value corresponding to the target pixel A, the overall filling process of the original image region to be estimated is completed.
[0086] As can be seen, by implementing the image depth estimation method described in the above embodiments, the electronic device can estimate the missing disparity data of the holed pixels in the image region to be estimated by utilizing the relatively accurate disparity data of pixels near the region edge, thereby improving the accuracy of the electronic device in estimating image depth information. Simultaneously, this estimation method not only reduces the computational load required to fill disparity holes, completing the image depth information estimation process quickly and accurately, but is also applicable to disparity holes of different sizes, further enhancing the accuracy and convenience of depth information estimation for electronic devices with binocular vision. Furthermore, by continuously utilizing pixels with estimable depth at the edges of disparity holes, the image region to be estimated containing disparity holes can be gradually filled, thereby completing the overall depth information calculation for the image region to be estimated and obtaining the depth map corresponding to the target image, facilitating further image processing operations based on depth information performed by the electronic device.
[0087] The methods in the embodiments of this application have been described in detail above. The apparatus in the embodiments of this application will be described below with reference to the accompanying drawings.
[0088] Please see Figure 8 , Figure 8 This is a modular schematic diagram of an image depth estimation device disclosed in an embodiment of this application. The image depth estimation device can be the aforementioned electronic device, or a device applied within the aforementioned electronic device. Figure 8 As shown, the image depth estimation device may include a first determining unit 801, a second determining unit 802, and a calculation unit 803, wherein:
[0089] The first determining unit 801 is used to determine at least one neighboring hole point corresponding to the target pixel point in the image region to be estimated; wherein the target pixel point is any pixel point in the image region to be estimated, and the initial disparity value corresponding to the target pixel point and the at least one neighboring hole point is 0, and the initial disparity value is the disparity value corresponding to the pixel point determined when the image is captured by a binocular device.
[0090] The second determining unit 802 is used to determine the target cavity point from at least one neighboring cavity point, and to determine at least one neighboring estimated point corresponding to the target cavity point;
[0091] The calculation unit 803 is used to calculate the estimated disparity value corresponding to the target hole point based on the initial disparity value corresponding to at least one neighboring estimated point, and the estimated disparity value is used to determine the image depth of the target hole point in the image region to be estimated.
[0092] As can be seen, by employing the image depth estimation device described in the above embodiments, the electronic device can estimate the missing disparity data of the holed pixels in the image region to be estimated by utilizing the relatively accurate disparity data of pixels near the region edge, thereby improving the accuracy of the electronic device in estimating image depth information. This not only reduces the computational load required to fill disparity holes and completes the image depth information estimation process quickly and accurately, but also applies to disparity holes of different sizes, further enhancing the accuracy and convenience of depth information estimation in electronic devices equipped with binocular vision.
[0093] In one embodiment, the first determining unit 801 can be specifically used to obtain, in the image region to be estimated, the pixel with an initial disparity value of 0 and the image distance from the target pixel along at least one first target direction, with the target pixel as the center, as the neighboring hole point of the target pixel in each of the first target directions.
[0094] Among them, the included angle between any two adjacent first target directions is equal.
[0095] In one embodiment, the second determining unit 802, when determining the target cavity point from at least one neighboring cavity point, may specifically include:
[0096] The target hole point is determined from the nearest neighboring hole point that has the greatest image distance to the target pixel point among the above at least one neighboring hole points.
[0097] Based on this, when determining at least one neighboring estimated point corresponding to the target cavity point, the second determining unit 802 may specifically include:
[0098] Centered on the target hole point, along at least one second target direction, obtain the pixel points whose image distance from the target hole point is equal to the target distance in each second target direction, and use them as the nearest estimated points of the target hole point in each second target direction.
[0099] The aforementioned target distance refers to the image distance between the target hole point and the target pixel point.
[0100] In one embodiment, the above-mentioned calculation unit 803 can be specifically used to calculate the mean of the initial disparity values corresponding to at least one neighboring estimated point, as the estimated disparity value corresponding to the target hole point.
[0101] In one embodiment, the image depth estimation device may further include a region determination unit (not shown). This region determination unit may be used to determine the initial disparity value corresponding to each pixel in the target image before the first determination unit 801 determines at least one neighboring hole point corresponding to the target pixel in the image region to be estimated, and add pixels with an initial disparity value of 0 to the image region to be estimated.
[0102] The aforementioned target image is an image obtained by taking a picture using a binocular device.
[0103] As can be seen, the image depth estimation device described in the above embodiments not only improves the accuracy of image depth estimation by electronic devices, but also reduces the computational load required to fill disparity holes, enabling rapid and accurate completion of the image depth estimation process. Furthermore, it is applicable to disparity holes of different sizes, further enhancing the accuracy and convenience of depth estimation for electronic devices equipped with binocular vision. In addition, by selecting neighboring estimation points relatively far from the target hole point for depth estimation, interference caused by drastic changes in disparity data at the edge of the disparity hole can be minimized, further improving the accuracy of depth estimation by electronic devices. Moreover, by continuously utilizing pixels with estimable depth at the edge of the disparity hole, the image region containing the disparity hole can be gradually filled, thereby completing the overall depth information calculation for the image region to be estimated and obtaining the depth map corresponding to the target image. This facilitates further image processing operations based on depth information performed by the electronic device.
[0104] Please see Figure 9 , Figure 9 This is a modular schematic diagram of an electronic device disclosed in an embodiment of this application. For example... Figure 9 As shown, the electronic device may include:
[0105] Memory 901 storing executable program code;
[0106] Processor 902 coupled to memory 901;
[0107] The processor 902 can call the executable program code stored in the memory 901 to execute all or part of the steps in any of the image depth estimation methods described in the above embodiments.
[0108] Furthermore, embodiments of this application disclose a computer-readable storage medium storing a computer program for electronic data interchange, wherein the computer program enables a computer to perform all or part of the steps in any of the image depth estimation methods described in the above embodiments.
[0109] Furthermore, this application further discloses a computer program product that, when run on a computer, enables the computer to perform all or part of the steps in any of the image depth estimation methods described in the above embodiments.
[0110] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. This program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.
[0111] The foregoing has provided a detailed description of an image depth estimation method, apparatus, electronic device, and storage medium disclosed in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. An image depth estimation method, characterized in that, include: In the image region to be estimated, with the target pixel as the center, along at least one first target direction, the pixel with an initial disparity value of 0 and the farthest image distance from the target pixel in each of the first target directions is obtained, and is taken as the neighboring hole point corresponding to the target pixel in each of the first target directions; wherein, the target pixel is any pixel in the image region to be estimated, and the initial disparity value corresponding to the target pixel and at least one neighboring hole point is 0, and the initial disparity value is the disparity value corresponding to the pixel determined when the image is captured by a binocular device; Among the at least one neighboring hole points, the one with the farthest image distance from the target pixel is determined as the target hole point. Using the target hole point as the center, along at least one second target direction, pixels whose image distance from the target hole point is equal to the target distance in each of the second target directions are obtained, and these pixels are used as the estimated nearest neighbor points of the target hole point in each of the second target directions; wherein, the target distance is the image distance between the target hole point and the target pixel. Based on the initial disparity value corresponding to at least one neighboring estimated point, the estimated disparity value corresponding to the target hole point is calculated, and the estimated disparity value is used to determine the image depth of the target hole point in the image region to be estimated. Centered on the target pixel, repeat the above steps until the estimated disparity value for each pixel in the image region to be estimated is calculated.
2. The method according to claim 1, characterized in that, The included angle between any two adjacent directions of the first target is equal.
3. The method according to claim 1, characterized in that, The step of calculating the estimated disparity value corresponding to the target hole point based on the initial disparity value corresponding to the at least one neighboring estimated point includes: Calculate the mean of the initial disparity values corresponding to the at least one neighboring estimated point, and use it as the estimated disparity value corresponding to the target hole point.
4. The method according to any one of claims 1 to 2, characterized in that, Before determining at least one neighboring hole point corresponding to the target pixel in the image region to be estimated, the method further includes: In the target image, the initial disparity value corresponding to each pixel is determined, and the pixels with an initial disparity value of 0 are added to the image region to be estimated; wherein, the target image is an image obtained by taking a picture through the binocular device.
5. An image depth estimation device, characterized in that, include: The first determining unit is configured to, in the image region to be estimated, take the target pixel as the center and along at least one first target direction respectively, obtain the pixel with an initial disparity value of 0 in each of the first target directions and the pixel with the farthest image distance from the target pixel, and use it as the neighboring hole point corresponding to the target pixel in each of the first target directions; wherein, the target pixel is any pixel in the image region to be estimated, the initial disparity value corresponding to the target pixel and at least one neighboring hole point is 0, and the initial disparity value is the disparity value corresponding to the pixel determined when the image is captured by a binocular device; The second determining unit is configured to determine the nearest neighboring hole point with the farthest image distance from the target pixel point among the at least one neighboring hole points as the target hole point, and, with the target hole point as the center, obtain, along at least one second target direction, pixels whose image distance from the target hole point is equal to the target distance in each of the second target directions, as the nearest estimated points of the target hole point in each of the second target directions; wherein, the target distance is the image distance between the target hole point and the target pixel point; The calculation unit is used to calculate the estimated disparity value corresponding to the target hole point based on the initial disparity value corresponding to at least one neighboring estimated point, and the estimated disparity value is used to determine the image depth of the target hole point in the image region to be estimated.
6. An electronic device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the method as described in any one of claims 1 to 4.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 4.