Depth estimation methods, apparatus, electronic devices, and computer-readable storage media

Abstract

This application relates to a depth estimation method, apparatus, electronic device, storage medium, and computer program product. The method includes: acquiring a first image and corresponding gyroscope data, and acquiring a second image and corresponding gyroscope data; shifting pixels in the first image based on the gyroscope data corresponding to the first image and the gyroscope data corresponding to the second image to generate at least two iso-depth disparity maps; each iso-depth disparity map corresponds to a different depth range, and the depth of each pixel in each iso-depth disparity map is within the same depth range; performing pixel matching between the second image and each of the at least two iso-depth disparity maps to determine the depth information of the first image. This method can improve the efficiency of depth estimation.

Description

Technical Field

[0001] This application relates to imaging technology, and in particular to a depth estimation method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Traditional camera modules acquire two-dimensional image information within the captured space, losing depth information (the distance between the subject and the camera module). However, depth information can enable more functions. For example, in portrait photography, the subject and background can be separated, and the background can be blurred to simulate the bokeh effect of a DSLR camera on a mobile phone camera module; or face recognition can be performed using depth information to improve the reliability of recognition and avoid being "misled" by flat images.

[0003] Traditional depth estimation methods typically involve acquiring multiple images using multiple camera modules and matching feature points from these images to estimate depth information.

[0004] However, traditional depth estimation methods suffer from low efficiency. Summary of the Invention

[0005] This application provides a depth estimation method, apparatus, electronic device, computer-readable storage medium, and computer program product that can improve the efficiency of depth estimation.

[0006] Firstly, this application provides a depth estimation method. The method includes:

[0007] Acquire the first image and the corresponding gyroscope data, and acquire the second image and the corresponding gyroscope data;

[0008] Based on the gyroscope data corresponding to the first image and the gyroscope data corresponding to the second image, the pixels in the first image are shifted to generate at least two equal-depth disparity maps; each equal-depth disparity map corresponds to a different depth range, and the depth of each pixel in each equal-depth disparity map is within the same depth range;

[0009] Pixel matching is performed on the second image and each of the at least two equal-depth disparity maps to determine the depth information of the first image.

[0010] Secondly, this application also provides a depth estimation apparatus. The apparatus includes:

[0011] The acquisition module is used to acquire the first image and the corresponding gyroscope data, and to acquire the second image and the corresponding gyroscope data;

[0012] The image generation module is used to shift the pixels in the first image based on the gyroscope data corresponding to the first image and the gyroscope data corresponding to the second image, and generate at least two equal-depth disparity maps; each equal-depth disparity map corresponds to a different depth range, and the depth of each pixel in each equal-depth disparity map is within the same depth range;

[0013] The depth determination module is used to perform pixel matching between the second image and each of the at least two equal-depth disparity maps to determine the depth information of the first image.

[0014] Thirdly, this application also provides an electronic device. The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0015] Acquire the first image and the corresponding gyroscope data, and acquire the second image and the corresponding gyroscope data;

[0016] Based on the gyroscope data corresponding to the first image and the gyroscope data corresponding to the second image, the pixels in the first image are shifted to generate at least two equal-depth disparity maps; each equal-depth disparity map corresponds to a different depth range, and the depth of each pixel in each equal-depth disparity map is within the same depth range;

[0017] Pixel matching is performed on the second image and each of the at least two equal-depth disparity maps to determine the depth information of the first image.

[0018] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0019] Acquire the first image and the corresponding gyroscope data, and acquire the second image and the corresponding gyroscope data;

[0020] Based on the gyroscope data corresponding to the first image and the gyroscope data corresponding to the second image, the pixels in the first image are shifted to generate at least two equal-depth disparity maps; each equal-depth disparity map corresponds to a different depth range, and the depth of each pixel in each equal-depth disparity map is within the same depth range;

[0021] Pixel matching is performed on the second image and each of the at least two equal-depth disparity maps to determine the depth information of the first image.

[0022] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0023] Acquire the first image and the corresponding gyroscope data, and acquire the second image and the corresponding gyroscope data;

[0024] Based on the gyroscope data corresponding to the first image and the gyroscope data corresponding to the second image, the pixels in the first image are shifted to generate at least two equal-depth disparity maps; each equal-depth disparity map corresponds to a different depth range, and the depth of each pixel in each equal-depth disparity map is within the same depth range;

[0025] Pixel matching is performed on the second image and each of the at least two equal-depth disparity maps to determine the depth information of the first image.

[0026] The aforementioned depth estimation method, apparatus, electronic device, computer-readable storage medium, and computer program product involve an electronic device acquiring a first image and corresponding gyroscope data, and acquiring a second image and corresponding gyroscope data. Based on the gyroscope data corresponding to the first and second images, pixels in the first image are shifted to generate at least two iso-depth disparity maps. Each iso-depth disparity map corresponds to a different depth range, and the depth of each pixel in each iso-depth disparity map is within the same depth range. By performing pixel matching between the second image and each of the at least two iso-depth disparity maps, the depth range corresponding to the matching region can be determined, thereby determining the depth information of the first image. This eliminates the need for feature point search, description, and matching operations, effectively reducing algorithm complexity and computational load, thus improving the efficiency of depth estimation. Simultaneously, it avoids depth estimation errors caused by feature point loss and mismatches, ensuring depth stability under different shooting scenarios. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 Here is a flowchart of a depth estimation method in one embodiment;

[0029] Figure 2 A flowchart for obtaining at least two equal-depth disparity maps in one embodiment;

[0030] Figure 3 A flowchart of the depth estimation method in another embodiment;

[0031] Figure 4 This is a flowchart illustrating how a target depth map is obtained based on a second image and at least two equal-depth disparity maps in one embodiment.

[0032] Figure 5 A flowchart of the depth estimation method in another embodiment;

[0033] Figure 6 This is a structural block diagram of a depth estimation device in one embodiment;

[0034] Figure 7 This is a diagram of the internal structure of an electronic device in one embodiment. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0036] In one embodiment, such as Figure 1 As shown, a depth estimation method is provided. This embodiment illustrates the application of this method to an electronic device, which can be a terminal or a server. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. The server can be a standalone server or a server cluster consisting of multiple servers.

[0037] In this embodiment, the depth estimation method includes the following steps 102 to 106:

[0038] Step 102: Obtain the first image and the corresponding gyroscope data, and obtain the second image and the corresponding gyroscope data.

[0039] The electronic device includes a single camera module, which in turn includes an image sensor. Specifically, a single camera module is a module containing a single camera, and the image sensor converts the light signals entering the electronic device into electrical signals and generates an image.

[0040] Electronic devices also include gyroscopes, through which gyroscope data can be acquired. This gyroscope data includes angular acceleration and linear acceleration.

[0041] Optionally, the electronic device captures a first image using a single camera module and simultaneously collects gyroscope data corresponding to the first image; it also captures a second image using a single camera module and simultaneously collects gyroscope data corresponding to the second image; wherein the first image is captured earlier than the second image.

[0042] It is understandable that the electronic device will shift during the shooting process, so there is a certain parallax between the first image and the second image acquired by the electronic device. That is, the first image is used as the reference frame and as the image referenced for the final output depth information, while the second image is used as the parallax frame relative to the first image.

[0043] Step 104: Based on the gyroscope data corresponding to the first image and the gyroscope data corresponding to the second image, the pixels in the first image are shifted to generate at least two equal-depth disparity maps; each equal-depth disparity map corresponds to a different depth range, and the depth of each pixel in each equal-depth disparity map is within the same depth range.

[0044] Depth refers to the distance between the subject being photographed and the single camera module, also known as depth distance. This can be understood as pre-setting all pixels in the equal-depth parallax map to be within the same depth range, or all pixels in the equal-depth parallax map to be at the same preset depth. The depth range can be set as needed. For example, the depth range can be a relatively small interval.

[0045] Optionally, each iso-depth disparity map corresponds to a different preset depth, and each pixel in each iso-depth disparity map has the same preset depth.

[0046] Each depth range corresponding to each isometric parallax map can be set according to the depth resolution required for the actual application. For example, the interval size of each depth range is 0.1m, and the maximum interval value between adjacent depth ranges differs by 0.5m.

[0047] Similarly, the preset depths corresponding to each iso-depth parallax map can be set as needed. For example, a preset depth can be set every 0.5m within a depth range of 1m to 5m.

[0048] Optionally, the electronic device determines the pixel displacement relationship between the first image and the second image based on the gyroscope data corresponding to the first image and the second image; based on the pixel displacement relationship, the pixels in the first image are displaced at least twice to generate at least two equal-depth disparity maps; wherein, each displacement generates one equal-depth disparity map.

[0049] It is understandable that during the process of capturing the first and second images, if the single camera module is displaced, there will also be a displacement between the pixels of the first image and the pixels of the second image. Therefore, based on the pixel displacement relationship between the first image and the second image, the pixels in the first image are displaced to generate at least two equal-depth parallax maps.

[0050] Step 106: Perform pixel matching between the second image and each of the at least two equal-depth disparity maps to determine the depth information of the first image.

[0051] Depth information is information that characterizes the depth distance between each pixel in the first image. For example, the depth information may be a depth map corresponding to the first image, which includes the depth information of the first image.

[0052] Optionally, the electronic device performs pixel matching between the second image and each of the at least two equal-depth parallax maps to determine the matching overlapping areas between the second image and each equal-depth parallax map. Then, the depth range corresponding to the overlapping area in each equal-depth parallax map, which is also the depth range of the corresponding area in the first image, can determine the depth information of each area in the first image.

[0053] The aforementioned depth estimation method involves an electronic device acquiring a first image and corresponding gyroscope data, as well as a second image and corresponding gyroscope data. Based on the gyroscope data corresponding to the first and second images, pixels in the first image are shifted to generate at least two iso-depth disparity maps. Each iso-depth disparity map corresponds to a different depth range, and the depth of each pixel in each iso-depth disparity map lies within the same depth range. By performing pixel matching between the second image and each of the at least two iso-depth disparity maps, the depth range corresponding to the matching region can be determined, thereby confirming the depth information of the first image. This eliminates the need for feature point search, description, and matching operations, effectively reducing algorithm complexity and computational load, thus improving the efficiency of depth estimation. Simultaneously, it avoids depth estimation errors caused by feature point loss and mismatches, ensuring depth stability under different shooting scenarios.

[0054] Furthermore, based on the data output by the gyroscope (accelerometer) that is usually equipped, and combined with the images captured by the camera module at multiple moments, the electronic device can obtain the depth information of the shooting scene, reducing the hardware cost required for depth estimation.

[0055] In one embodiment, based on gyroscope data corresponding to a first image and gyroscope data corresponding to a second image, the pixels in the first image are shifted to generate at least two equal-depth disparity maps, including: determining the spatial positional relationship between the image sensor when capturing the first image and the second image based on the gyroscope data corresponding to the first image and the second image; and shifting the pixels of the first image based on the spatial positional relationship to generate at least two equal-depth disparity maps.

[0056] The spatial relationship between the image sensor and the first and second images can specifically include the baseline (optical center distance) of the camera module between the first and second images, the angle between the optical axes of the module, and the movement distance.

[0057] Optionally, determining the spatial positional relationship of the image sensor between the first and second images based on the gyroscope data corresponding to the first and second images includes: determining the motion information of the gyroscope between the gyroscope data corresponding to the first and second images based on the gyroscope data corresponding to the first and second images; converting the motion information of the gyroscope into the motion information of the image sensor according to the position conversion relationship between the gyroscope and the image sensor; and determining the spatial positional relationship of the image sensor between the first and second images based on the motion information of the image sensor.

[0058] Motion information includes at least one of the following: distance traveled, rotation angle, linear velocity, and rotational speed.

[0059] Gyroscope data includes linear acceleration and angular acceleration. Electronic devices can obtain the linear velocity and angular velocity of the gyroscope by integrating the gyroscope data once. Then, by integrating the linear velocity and angular velocity again, the distance traveled and the rotation angle of the gyroscope can be obtained.

[0060] Optionally, the electronic device inputs the gyroscope data into a filter for signal processing. The filter functions to reduce noise, remove or add the corresponding DC bias signal. The filtered signal is integrated once by an integrator to obtain the gyroscope's own angular velocity and linear velocity. After another integration, the gyroscope's movement distance and rotation angle are obtained.

[0061] Optionally, the electronic device integrates the gyroscope data corresponding to the first image twice to obtain the motion information of the gyroscope during the first image capture process, and integrates the gyroscope data corresponding to the second image twice to obtain the motion information of the gyroscope during the second image capture process; adding the motion information of the gyroscope during the first image capture process and the motion information of the gyroscope during the second image capture process, the motion information between the gyroscope data corresponding to the first image and the gyroscope data corresponding to the second image can be obtained.

[0062] It is understandable that, since the gyroscope and image sensor are not located in the same spatial position, the position transformation relationship between the gyroscope and image sensor is obtained. Based on this relationship, the motion information of the gyroscope is converted into motion information of the image sensor. This position transformation relationship can be pre-calibrated; for example, it can be a transformation matrix H, which is pre-calibrated by the electronic device based on the positions of the gyroscope and image sensor. The motion information of the image sensor includes the distance traveled and the rotation angle.

[0063] In one embodiment, such as Figure 2 As shown, the electronic device inputs the gyroscope data corresponding to the first image and the gyroscope data corresponding to the second image into filters for signal processing. The filters function to reduce noise and remove or add corresponding DC bias signals. The filtered signals are integrated once by an integrator to obtain the angular velocity and linear velocity of the gyroscope corresponding to the first image and the angular velocity and linear velocity of the gyroscope corresponding to the second image. The angular velocity and linear velocity are then input into the integrator and integrated once to obtain the movement distance and rotation angle of the gyroscope corresponding to the first image and the movement distance and rotation angle of the gyroscope corresponding to the second image.

[0064] The electronic device, according to a transformation matrix, converts the movement distance and rotation angle of the gyroscope corresponding to the first image into the movement distance and rotation angle of the image sensor when capturing the first image, and the same conversion applies to the movement distance and rotation angle of the gyroscope corresponding to the second image. Based on the movement distance and rotation angle of the image sensor when capturing the first and second images, the spatial positional relationship between the image sensor and the first and second images can be determined. Based on this spatial relationship, the pixels of the first image are shifted to generate at least two equal-depth parallax maps.

[0065] In other alternative implementations, the electronic device can directly determine the spatial positional relationship between the image sensor when capturing the first image and the second image based on the motion relationship between the gyroscope data corresponding to the first image and the gyroscope data corresponding to the second image.

[0066] In this embodiment, the electronic device determines the spatial positional relationship between the image sensor and the first and second images based on the gyroscope data corresponding to the first and second images. Based on this spatial positional relationship, the displacement of the image sensor between the first and second images can be accurately determined, thereby shifting the pixels of the first image and accurately generating at least two equal-depth parallax maps.

[0067] In one embodiment, such as Figure 3 The present invention provides another depth estimation method, including the following steps 302 to 310:

[0068] Step 302: Obtain the first image and the corresponding gyroscope data, and obtain the second image and the corresponding gyroscope data.

[0069] Step 304: Based on the gyroscope data corresponding to the first image and the gyroscope data corresponding to the second image, determine the spatial positional relationship between the image sensor when capturing the first image and the second image.

[0070] Step 306: Based on spatial location relationships, determine the pixel movement distance corresponding to each depth range in at least two depth ranges.

[0071] Optionally, the electronic device acquires the correspondence between depth range and pixel movement distance under the spatial position relationship of the image sensor; based on the correspondence, it finds the pixel movement distance corresponding to each of at least two depth ranges.

[0072] For example, when the gyroscope detects that the electronic device moves to the left and determines that the spatial position relationship between the image sensor when capturing the first image and the second image is that the position of the image sensor when capturing the first image is to the right of the position when capturing the second image by a distance x, then under this spatial position relationship, the pixel movement distance corresponding to a depth distance of 5m is 2 pixels, and the pixel movement distance corresponding to a depth distance of 1m is 10 pixels.

[0073] Understandably, the smaller the depth distance, the greater the deviation in the field of view when the electronic device is displaced. That is, under this spatial relationship, the electronic device moves the pixels of the first image 2 pixels to the left to obtain a parallax map with a depth distance of 5m, and moves the pixels of the first image 10 pixels to the left to obtain a parallax map with a depth distance of 1m.

[0074] Step 308: For each depth range, the pixel movement distance corresponding to the pixel displacement depth range of the first image is used to obtain an equal depth disparity map; each equal depth disparity map corresponds to a different depth range, and the depth of each pixel in each equal depth disparity map is within the same depth range.

[0075] For example, a depth range is set at 1m intervals. Each depth range includes (0.5-1.0), (1.5-2.0), etc. The pixel movement distance corresponding to the depth range (0.5-1.0) is 10 pixels, and the pixel movement distance corresponding to the depth range (1.5-2.0) is 5 pixels. Then, for the depth range (0.5-1.0), the pixels of the first image of the electronic device are shifted by 10 pixels to obtain an equal-depth disparity map where the pixel depth is all in the range (0.5-1.0). The pixels of the first image are shifted by 5 pixels to obtain an equal-depth disparity map where the pixel depth is all in the range (1.5-2.0). This process is repeated to obtain at least two equal-depth disparity maps.

[0076] Step 310: Perform pixel matching between the second image and each of the at least two equal-depth disparity maps to determine the depth information of the first image.

[0077] In this embodiment, based on the spatial positional relationship between the first image and the second image captured by the image sensor, the pixel movement distance corresponding to each depth range in at least two depth ranges can be accurately determined. Thus, for each depth range, the pixel movement distance corresponding to the pixel displacement depth range of the first image is used to obtain an equal-depth disparity map.

[0078] In one embodiment, performing pixel matching between the second image and each of the at least two iso-depth disparity maps to determine the depth information of the first image includes: performing pixel matching between the second image and each of the at least two iso-depth disparity maps to determine the overlapping area between the second image and each iso-depth disparity map; and determining the depth information of the first image based on the overlapping area in each iso-depth disparity map.

[0079] Optionally, the electronic device subtracts each of the two iso-depth disparity maps from the second image, and identifies pixels whose difference is less than a preset difference threshold as pixels in the overlapping region, thus determining the overlapping region between the second image and each iso-depth disparity map. The preset difference threshold can be set as needed.

[0080] It is understandable that there is an overlapping area between the second image and the iso-depth disparity map, that is, the overlapping area in the second image and the overlapping area in the iso-depth disparity map belong to the same depth range or are at the same preset depth.

[0081] Optionally, the depth information of the first image is determined based on the overlapping regions in each equal-depth disparity map, including: obtaining the depth information corresponding to each overlapping region; using the depth information corresponding to each overlapping region as the depth information of the corresponding region in the first image to generate a target depth map of the first image.

[0082] It is understandable that the overlapping area between the second image and the iso-depth disparity map can determine the depth range of the iso-depth disparity map, which is obtained by simulating the displacement of the first image to obtain the second image, that is, by displacing the pixels in the first image. Therefore, the depth information of the overlapping area in the iso-depth disparity map can be used as the depth information of the corresponding area in the first image to generate the target depth map of the first image.

[0083] Optionally, the depth information corresponding to each overlapping region is used as the depth information of the corresponding region in the first image to generate a target depth map of the first image, including: using the depth information corresponding to each overlapping region as the depth information of the corresponding region in the first image to obtain an intermediate depth map; performing morphological processing or filtering processing on the depth information of each region in the intermediate depth map to obtain the target depth map.

[0084] Morphological processing can include filling areas lacking depth information, as well as techniques such as erosion and dilation. Filtering processing can involve selecting regions containing multiple depth parameters.

[0085] Alternatively, electronic devices can also use neural networks to perform morphological processing or filtering on intermediate depth maps to obtain target depth maps.

[0086] In this embodiment, the electronic device uses the depth information corresponding to each overlapping region as the depth information of the corresponding region in the first image to obtain an intermediate depth map. Then, morphological processing or filtering processing is performed on the depth information of each region in the intermediate depth map to obtain a more accurate target depth map.

[0087] In one embodiment, such as Figure 4 As shown, the electronic device performs pixel matching between the second image and the iso-depth disparity map of depth 1 to determine the overlapping area between the second image and the iso-depth disparity map of depth 1. It then performs pixel matching between the second image and the iso-depth disparity map of depth 2 to determine the overlapping area between the second image and the iso-depth disparity map of depth 2. The overlapping areas of the iso-depth disparity maps of depth 1 and depth 2 are superimposed to obtain an intermediate depth map. Image processing is then performed on the intermediate depth map, such as morphological processing or filtering of the depth information of each region in the intermediate depth map, to obtain the target depth map.

[0088] In one embodiment, such as Figure 5As shown, the electronic device acquires a first image and corresponding gyroscope data, and acquires a second image and corresponding gyroscope data after the electronic device moves; based on the gyroscope data corresponding to the first image and the second image, the spatial positional relationship between the image sensor and the captured first and second images is determined; based on the spatial positional relationship, the pixels of the first image are shifted to generate at least two equal-depth disparity maps; the overlapping regions of the second image and each of the at least two equal-depth disparity maps are matched to determine the overlapping regions between the second image and each equal-depth disparity map; the overlapping regions in each equal-depth disparity map are superimposed to obtain an intermediate depth map; image processing is performed on the intermediate depth map, such as morphological processing or filtering of the depth information of each region in the intermediate depth map, to obtain a target depth map.

[0089] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0090] Based on the same inventive concept, this application also provides a depth estimation apparatus for implementing the depth estimation method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more depth estimation apparatus embodiments provided below can be found in the limitations of the depth estimation method described above, and will not be repeated here.

[0091] In one embodiment, such as Figure 6 As shown, a depth estimation device is provided, including: an acquisition module 602, an image generation module 604, and a depth determination module 606, wherein:

[0092] The acquisition module 602 is used to acquire the first image and the corresponding gyroscope data, and to acquire the second image and the corresponding gyroscope data.

[0093] The image generation module 604 is used to shift the pixels in the first image based on the gyroscope data corresponding to the first image and the gyroscope data corresponding to the second image to generate at least two equal-depth disparity maps; each equal-depth disparity map corresponds to a different depth range, and the depth of each pixel in each equal-depth disparity map is within the same depth range.

[0094] The depth determination module 606 is used to perform pixel matching between the second image and each of the at least two equal-depth disparity maps to determine the depth information of the first image.

[0095] The aforementioned depth estimation device involves an electronic device acquiring a first image and corresponding gyroscope data, as well as a second image and corresponding gyroscope data. Based on the gyroscope data corresponding to the first and second images, the pixels in the first image are shifted to generate at least two iso-depth disparity maps. Each iso-depth disparity map corresponds to a different depth range, and the depth of each pixel in each iso-depth disparity map is within the same depth range. By performing pixel matching between the second image and each of the at least two iso-depth disparity maps, the depth range corresponding to the matching region can be determined, thereby determining the depth information of the first image. This eliminates the need for feature point search, description, and matching operations, effectively reducing the complexity and computational load of the algorithm and improving the efficiency of depth estimation.

[0096] In one embodiment, the image generation module 604 is further configured to determine the spatial positional relationship between the image sensor capturing the first image and the second image based on the gyroscope data corresponding to the first image and the second image; and to shift the pixels of the first image based on the spatial positional relationship to generate at least two equal-depth parallax maps.

[0097] In one embodiment, the image generation module 604 is further configured to determine the pixel movement distance corresponding to each depth range in at least two depth ranges based on spatial positional relationships; and to obtain an equal-depth disparity map by taking the pixel movement distance corresponding to the pixel displacement depth range of the first image for each depth range.

[0098] In one embodiment, the image generation module 604 is further configured to determine the motion information of the gyroscope between the gyroscope data corresponding to the first image and the gyroscope data corresponding to the second image based on the gyroscope data corresponding to the first image and the gyroscope data corresponding to the second image; convert the motion information of the gyroscope into the motion information of the image sensor according to the position conversion relationship between the gyroscope and the image sensor; and determine the spatial positional relationship of the image sensor between capturing the first image and the second image based on the motion information of the image sensor.

[0099] In one embodiment, the depth determination module 606 is further configured to perform pixel matching between the second image and each of the at least two equal-depth disparity maps to determine the overlapping area between the second image and each equal-depth disparity map; and to determine the depth information of the first image based on the overlapping area in each equal-depth disparity map.

[0100] In one embodiment, the depth determination module 606 is further configured to obtain depth information corresponding to each overlapping region; and use the depth information corresponding to each overlapping region as the depth information of the corresponding region in the first image to generate a target depth map of the first image.

[0101] In one embodiment, the depth determination module 606 is further configured to use the depth information corresponding to each overlapping region as the depth information of the corresponding region in the first image to obtain an intermediate depth map; and to perform morphological processing or filtering processing on the depth information of each region in the intermediate depth map to obtain a target depth map.

[0102] Each module in the aforementioned depth estimation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the electronic device in hardware form or independent of it, or stored in the memory of the electronic device in software form, so that the processor can call and execute the operations corresponding to each module.

[0103] In one embodiment, an electronic device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7As shown, this electronic device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a depth estimation method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the electronic device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the electronic device, or external keyboards, touchpads, or mice, etc.

[0104] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0105] This application also provides a computer-readable storage medium. One or more non-volatile computer-readable storage media containing computer-executable instructions, which, when executed by one or more processors, cause the processors to perform the steps of a depth estimation method.

[0106] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform a depth estimation method.

[0107] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0108] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0109] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0110] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A depth estimation method, characterized by, include: Acquire the first image and the corresponding gyroscope data, and acquire the second image and the corresponding gyroscope data; Based on the gyroscope data corresponding to the first image and the gyroscope data corresponding to the second image, the spatial positional relationship between the image sensor when capturing the first image and the second image is determined; Based on the spatial relationship, the pixel movement distance corresponding to each of the at least two depth ranges is determined; for each depth range, the pixels of the first image are shifted by the pixel movement distance corresponding to the depth range to obtain an iso-depth disparity map; each iso-depth disparity map corresponds to a different depth range, and the depth of each pixel in each iso-depth disparity map is within the same depth range; Pixel matching is performed on the second image and each of the at least two iso-depth disparity maps to determine the overlapping area between the second image and each iso-depth disparity map; based on the overlapping area in each iso-depth disparity map, the depth information of the first image is determined.

2. The method of claim 1, wherein, Determining the spatial positional relationship between the image sensor and the second image based on the gyroscope data corresponding to the first image and the second image includes: Based on the gyroscope data corresponding to the first image and the gyroscope data corresponding to the second image, determine the motion information of the gyroscope between the gyroscope data corresponding to the first image and the gyroscope data corresponding to the second image; According to the position conversion relationship between the gyroscope and the image sensor, the motion information of the gyroscope is converted into the motion information of the image sensor; Based on the motion information of the image sensor, the spatial positional relationship between the image sensor and the captured first image and second image is determined.

3. The method of claim 1, wherein, Determining the depth information of the first image based on the overlapping regions in each equal-depth disparity map includes: Obtain the depth information corresponding to each overlapping region; The depth information corresponding to each overlapping region is used as the depth information of the corresponding region in the first image to generate the target depth map of the first image.

4. The method of claim 3, wherein, The step of using the depth information corresponding to each overlapping region as the depth information of the corresponding region in the first image to generate a target depth map of the first image includes: The depth information corresponding to each overlapping region is used as the depth information of the corresponding region in the first image to obtain an intermediate depth map. Morphological processing or filtering is performed on the depth information of each region in the intermediate depth map to obtain the target depth map.

5. A depth estimation device, characterized in that, include: The acquisition module is used to acquire the first image and the corresponding gyroscope data, and to acquire the second image and the corresponding gyroscope data; The image generation module is used to determine the spatial positional relationship between the image sensor when capturing the first image and the second image based on the gyroscope data corresponding to the first image and the gyroscope data corresponding to the second image; Based on the spatial relationship, the pixel movement distance corresponding to each of the at least two depth ranges is determined; for each depth range, the pixels of the first image are shifted by the pixel movement distance corresponding to the depth range to obtain an iso-depth disparity map; each iso-depth disparity map corresponds to a different depth range, and the depth of each pixel in each iso-depth disparity map is within the same depth range; The depth determination module is used to perform pixel matching between the second image and each of the at least two iso-depth disparity maps to determine the overlapping area between the second image and each iso-depth disparity map; and to determine the depth information of the first image based on the overlapping area in each iso-depth disparity map.

6. The apparatus according to claim 5, characterized in that, The image generation module is further configured to determine the motion information of the gyroscope between the gyroscope data corresponding to the first image and the gyroscope data corresponding to the second image, based on the gyroscope data corresponding to the first image and the gyroscope data corresponding to the second image; convert the motion information of the gyroscope into the motion information of the image sensor according to the position conversion relationship between the gyroscope and the image sensor; and determine the spatial positional relationship of the image sensor between capturing the first image and the second image based on the motion information of the image sensor.

7. The apparatus according to claim 5, characterized in that, The depth determination module is also used to obtain the depth information corresponding to each overlapping region; and to use the depth information corresponding to each overlapping region as the depth information of the corresponding region in the first image to generate the target depth map of the first image.

8. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the processor, the processor performs the steps of the depth estimation method as described in any one of claims 1 to 4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 4.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images