Image processing method and device, electronic equipment and readable storage medium

Abstract

The application discloses an image processing method and device, electronic equipment and a readable storage medium, and belongs to the technical field of image processing. The method comprises the following steps: acquiring a first depth image, the first depth image comprising M depth points, one depth point corresponding to one first depth value; determining N first disparity values corresponding to N first pixel points according to the M first depth values; determining second disparity information of each first pixel point according to the first disparity value and the first disparity information corresponding to each first pixel point, the first disparity information being disparity information between the first pixel point and a second pixel point corresponding thereto, the second pixel point being a pixel point in a second field of view image, the second field of view image and the first field of view image being images of different fields of view shot under the same scene; determining second depth values of the first pixel points according to each second disparity information, and generating a second depth map according to the N second depth values.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, specifically relating to an image processing method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] With the development of three-dimensional (3D) technology, depth information of a scene is required in applications such as stereoscopic display, machine vision, and satellite remote sensing.

[0003] In related technologies, depth information of a scene can be obtained through binocular ranging. Specifically, two images with slight horizontal parallax are captured simultaneously by a binocular camera to obtain a binocular stereo image. Utilizing the parallax characteristics of human binocular vision, a stereo matching algorithm is used to calculate the three-dimensional geometric information of the object being measured from the multiple images acquired by the binocular camera, thus obtaining the depth value of the target object.

[0004] However, since stereo matching algorithms are only applicable to generating depth images of target objects by calculating depth values ​​after inputting images, the depth values ​​obtained by electronic devices after converting disparity values ​​may have corresponding errors, resulting in low accuracy of the calculated depth values. Summary of the Invention

[0005] The purpose of this application is to provide an image processing method, apparatus, electronic device, and readable storage medium that can improve the accuracy of depth values ​​in depth images.

[0006] In a first aspect, embodiments of this application provide an image processing method, which includes: acquiring a first depth image, the first depth image including M depth points, each depth point corresponding to a first depth value, where M is a positive integer; determining N first disparity values ​​corresponding to N first pixels based on the M first depth values, where the N first pixels are all pixels in a first field of view image, and N is a positive integer greater than M; determining second disparity information for each first pixel based on the first disparity value and first disparity information corresponding to each first pixel, where the first disparity information is the disparity information between the first pixel and its corresponding second pixel, and the second pixel is a pixel in a second field of view image, where the second field of view image and the first field of view image are images of different fields of view captured in the same scene; determining a second depth value for each first pixel based on each second disparity information, and generating a second depth map based on the N second depth values.

[0007] Secondly, embodiments of this application provide an image processing apparatus, comprising: an acquisition module, a processing module, and a generation module; the acquisition module is configured to acquire a first depth image, the first depth image including M depth points, each depth point corresponding to a first depth value, where M is a positive integer; the processing module is configured to determine N first disparity values ​​corresponding to N first pixels based on the M first depth values ​​acquired by the acquisition module, where the N first pixels are all pixels in a first field of view image, and N is a positive integer greater than M; the processing module is further configured to determine second disparity information for each first pixel based on the first disparity value and first disparity information corresponding to each first pixel, where the first disparity information is the disparity information between the first pixel and its corresponding second pixel, and the second pixel is a pixel in a second field of view image, where the second field of view image and the first field of view image are images of different fields of view captured in the same scene; the processing module is further configured to determine a second depth value based on each second disparity information; the generation module is configured to generate a second depth map based on the N second depth values.

[0008] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0009] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0010] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0011] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0012] In this embodiment, a first depth image is acquired, which includes M depth points, each depth point corresponding to a first depth value, where M is a positive integer. Based on the M first depth values, N first disparity values ​​corresponding to N first pixels are determined, where the N first pixels are all pixels in the first field of view image, and N is a positive integer greater than M. Based on the first disparity value and first disparity information corresponding to each first pixel, second disparity information for each first pixel is determined, where the first disparity information is the disparity information between the first pixel and its corresponding second pixel, and the second pixel is a pixel in the second field of view image. The second field of view image and the first field of view image are images of different fields of view captured in the same scene. Based on each second disparity information, a second depth value for each first pixel is determined, and a second depth map is generated based on the N second depth values. In this scheme, based on the depth value in a highly accurate depth image, N highly accurate disparity values ​​corresponding to N first pixels in the first field of view are determined. Then, these highly accurate disparity values ​​are fused and calculated with the first disparity information to correct each other, so as to obtain highly accurate second disparity information. The second depth value is determined based on the second disparity information, thereby improving the accuracy of the depth value. In this way, a depth map that is more consistent with the actual three-dimensional object can be generated, that is, the accuracy of the generated depth map is improved. Attached Figure Description

[0013] Figure 1 This is one of the schematic diagrams of an image processing method provided in an embodiment of this application;

[0014] Figure 2 This is a second schematic diagram of an image processing method provided in an embodiment of this application;

[0015] Figure 3 This is a third schematic diagram of an image processing method provided in an embodiment of this application;

[0016] Figure 4 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0017] Figure 5 This is one of the hardware structure diagrams of an electronic device provided in the embodiments of this application;

[0018] Figure 6 This is a second schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0020] The following explains the technical terms used in the embodiments of this application:

[0021] 1) Direct-Time of Flight (d-ToF):

[0022] In this embodiment, d-ToF refers to the direct time-of-flight method. d-ToF technology directly measures the time difference between the emission and reception of a light pulse. It uses single-photon avalanche diodes (SPADs) to achieve high-sensitivity light detection and employs time-correlated single-photon counting (TCSPC) to achieve picosecond-level time accuracy. d-ToF transmits discrete laser pulses, achieving an ultra-low duty cycle. Compared to indirect time-of-flight (i-ToF), it is more power-efficient and has a faster imaging speed, but it has higher technical barriers and requires more sophisticated hardware. Because d-ToF has advantages over i-ToF in terms of rapid information acquisition, interference resistance, and image clarity, and with the continuous maturation and improvement of algorithms and hardware, d-ToF is expected to become the mainstream solution for depth-sensing imaging technology.

[0023] 2) d-ToF sensor: Its principle is to use short pulse lasers at the nanosecond or even picosecond level to emit directly at the transmitting end and then quickly receive the reflected lasers. The sensor obtains ranging information by measuring the time of flight.

[0024] 3) Depth map: Defines the distance component values ​​along the optical axis from the optical center of the camera to the object point. The depth map can convert pixel coordinates (u,v) into spatial three-dimensional point coordinates (x,y,z).

[0025] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0026] The terms "at least one," "at least one of," etc., used in the specification and claims of this application refer to any one, any two, or a combination of two or more of the included items. For example, "at least one of a, b, and c" can mean: "a," "b," "c," "a and b," "a and c," "b and c," and "a, b, and c," where a, b, and c can be single or multiple. Similarly, "at least two" refers to two or more items, and its meaning is similar to that of "at least one."

[0027] It should be noted that the image processing method provided in this application can be executed by electronic devices such as mobile phones, tablets, laptops, PDAs, and in-vehicle electronic devices. Some embodiments of this application use electronic devices as the executing entity to illustrate the image processing method provided in this application.

[0028] The image processing method, apparatus, electronic device, and readable storage medium provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0029] Depth estimation algorithms have important applications in 3D reconstruction and augmented / mixed reality. Depth maps can assist in MR spatial perception and virtual-real interaction. Accurate depth results can also help achieve occlusion effects between virtual and real environments, thus reflecting convincing 3D information about the real environment and enabling people to obtain a natural spatial perception. In recent years, with the development of 3D technology, applications such as stereoscopic displays, machine vision, and satellite remote sensing all require the acquisition of scene depth information. Currently, the principles for acquiring depth information can be divided into three types: binocular ranging, structured light, and time-of-flight (TOF). Among them, TOF is further divided into direct time-of-flight (dTOF) and indirect time-of-flight (iTOF) based on different ranging principles.

[0030] Due to its advantages of high measurement accuracy, compact structure, fast response speed, and low power consumption, dTOF sensors are considered an ideal choice for next-generation 3D sensing devices. In related technologies, the process of acquiring depth information using a dTOF sensor involves: the transmitter emitting a modulated pulse of light with concentrated energy, while the receiver activates within the same pulse cycle to receive photons. After exposure through multiple pulse cycles, the receiver receives a sufficient number of photons, thus creating a histogram showing the distribution of photons received by each pixel at different times within a pulse cycle. Based on this histogram, the depth information of the target can be recovered.

[0031] However, on the one hand, the area covered by dTOF points for Vision Pro glasses is much smaller than the field of view of a color camera, thus limiting the spatial reconstruction range. On the other hand, due to manufacturing limitations, the number of laser points emitted by the dTOF sensor is limited, resulting in low spatial resolution of the acquired dTOF data. The sparse 3D points obtained through dTOF typically only yield a relatively sparse depth map, making it difficult to directly apply in some MR scenarios (such as 3D reconstruction occlusion and collision).

[0032] To address the aforementioned problems, those skilled in the art typically employ the following existing technologies:

[0033] Existing technology 1:

[0034] By setting two different regional location information to obtain the depth value of each pixel, this method achieves low-cost and efficient dynamic adjustment of angular resolution, solving the problem of insufficient angular resolution in related technologies and enabling the acquisition of depth points at different angular resolutions. Although this method improves the angular resolution of the point cloud to some extent, it still cannot obtain pixel-by-pixel point cloud and depth information. Furthermore, the limited frame rate of dTOF makes it difficult to acquire high-frequency depth information, thus limiting its application in practical AR / MR applications.

[0035] Existing technology 2:

[0036] By acquiring sparse pixel depth images of faces from different angles and fusing multiple pixel depth images to obtain a dense pixel depth map, face recognition based on this map can improve accuracy to some extent. Although this method can obtain a dense depth map, it is difficult to require users to capture images from multiple angles and then fuse them to generate a higher-precision depth map in AR / MR applications, thus limiting its effectiveness.

[0037] Existing technology three:

[0038] Depth densification is achieved by enhancing low-resolution dToF depth using high-resolution RGB images. However, for AR / MR applications, the field of view of a fisheye RGB camera is usually much larger than that of the dToF point cloud. This results in most areas of the RGB image not being covered by the dToF point cloud, leading to a decrease in depth accuracy. Furthermore, the dToF frame rate limitation makes it difficult to meet the requirements for outputting high-frame-rate depth.

[0039] However, in the image processing method provided in this application embodiment, a first depth image is obtained, which includes M first depth values, where M is a positive integer; based on the M first depth values, N first disparity values ​​corresponding to N first pixels are determined, where the first pixel is a pixel in the first field of view image corresponding to the first depth image, and N is a positive integer greater than M; based on the first disparity value and first disparity information corresponding to each first pixel, a second disparity information is determined, where the first disparity information is the disparity information between the first pixel and its corresponding second pixel, and the second pixel is a pixel in the second field of view image; based on each second disparity information, a second depth value is determined, and a second depth map is generated based on the N second depth values. In this scheme, based on the depth value in a highly accurate depth image, N highly accurate disparity values ​​corresponding to N first pixels in the first field of view are determined. Then, these highly accurate disparity values ​​are fused and calculated with the first disparity information to correct each other, so as to obtain highly accurate second disparity information. The second depth value is determined based on the second disparity information, thereby improving the accuracy of the depth value. In this way, a depth map that is more consistent with the actual three-dimensional object can be generated, that is, the accuracy of the generated depth map is improved.

[0040] The image processing method provided in this embodiment can be executed by an image processing device, which can be an electronic device, or a control module or processing module within the electronic device. The following description uses an electronic device as an example to illustrate the technical solution provided in this application embodiment.

[0041] This application provides an image processing method. Figure 1 A flowchart illustrating an image processing method provided in an embodiment of this application is shown, which can be applied to electronic devices. Figure 1 As shown, the image processing method provided in this application embodiment may include the following steps 201 to 204.

[0042] Step 201: The electronic device acquires the first depth image.

[0043] In some embodiments of this application, the first depth image includes M depth points, where M is a positive integer.

[0044] In some embodiments of this application, a depth point corresponds to a first depth value.

[0045] In some embodiments of this application, the first depth image described above is acquired by a dTOF sensor.

[0046] For example, the first depth image described above is generated based on the depth value corresponding to the laser time of flight obtained by the dTOF sensor.

[0047] For example, the aforementioned depth points can be sparse point cloud points in a first depth image.

[0048] For example, each point cloud point represents a depth value at a location.

[0049] In other words, each first depth value is the depth value of a point cloud point in the first depth image.

[0050] Step 202: The electronic device determines the N first disparity values ​​corresponding to the N first pixels based on the M first depth values.

[0051] In some embodiments of this application, the aforementioned N first pixels are all pixels in the first field of view image, where N is a positive integer greater than M.

[0052] It is understandable that the first depth image and the first field of view image corresponding to the first depth image are images of the same scene.

[0053] In some embodiments of this application, the first field-of-view image is a three-dimensional stereoscopic image converted from one of the RGB images obtained by a binocular camera.

[0054] In some embodiments of this application, the above-mentioned binocular RGB image includes a left-eye RGB image and a right-eye RGB image.

[0055] For example, the first field of view image mentioned above can be a left-eye stereo image obtained by stereo correction of a left-eye RGB image, or a left-eye stereo image obtained by stereo correction of a right-eye RGB image.

[0056] It should be noted that the field of view image corresponding to the left eye's RGB image is usually processed as the main image.

[0057] Optionally, in some embodiments of this application, before determining the M depth values ​​in the first field-of-view image, the electronic device also needs to perform a stereo correction operation on the image to be processed to obtain the first field-of-view image. For example, the image to be processed can be rotated using a rotation matrix, transforming the image to be processed into a space parallel to the optical axis.

[0058] In some embodiments of this application, the electronic device projects M point cloud points corresponding to M depth values ​​(i.e., the aforementioned depth points) onto a first field-of-view image, determines M first pixels based on the positions of the point cloud points, and then calculates the first disparity values ​​of the M first pixels based on the depth values ​​of the point cloud points corresponding to the M first pixels, thus obtaining M first disparity values. Since the M point cloud points cannot completely cover all the first pixels in the first field-of-view image, the electronic device determines the disparity values ​​of the first pixels other than the M first pixels as preset disparity values, thereby obtaining the first disparity values ​​of the remaining first pixels. Finally, the first disparity values ​​corresponding to the M first pixels and the first disparity values ​​corresponding to the remaining first pixels are determined as N first disparity values ​​for N first pixels.

[0059] For example, the preset disparity value can be 0 or any value.

[0060] For example, the electronic device can calculate the disparity value of a pixel based on the pixel's depth value and the camera focal length and binocular camera baseline corresponding to the first field of view image.

[0061] Optionally, in some embodiments of this application, step 202 specifically includes steps 202a to 202d:

[0062] Step 202a: The electronic device projects M depth points onto the first field of view image to determine M first pixel points.

[0063] In some embodiments of this application, the above-mentioned M first pixels correspond one-to-one with M depth points.

[0064] In some embodiments of this application, the aforementioned M first pixels are M pixels out of N first pixels in the aforementioned first field of view image.

[0065] Example 1: Suppose that the first depth image includes point cloud points A, B, and C, which are the aforementioned depth points, and the first field of view image includes first pixel points A, B, C, D, and E. The point cloud points in the first depth image are projected onto the first pixel points in the first field of view image, that is, point cloud point A corresponds to first pixel point A, point cloud point B corresponds to first pixel point B, and point cloud point C corresponds to first pixel point C.

[0066] Step 202b: The electronic device determines the first disparity value of the first pixel corresponding to each depth point based on the first depth value of each depth point, and obtains M first disparity values.

[0067] In some embodiments of this application, the electronic device calculates the first disparity value of the first pixel corresponding to the depth point based on the depth value of a depth point, the camera focal length in the first field of view image, and the baseline of the binocular camera.

[0068] For example, the electronic device uses formula (1) to calculate the first disparity value of the first pixel.

[0069] For example, the above formula (1) is:

[0070]

[0071] Where f represents the camera focal length, B represents the binocular camera baseline, z represents the depth value, and d is the first parallax value.

[0072] Step 202c: The electronic device will set the preset disparity value as NM first disparity values ​​corresponding to NM first pixels.

[0073] Step 202d: Determine NM first disparity values ​​and M first disparity values ​​as N first disparity values.

[0074] In some embodiments of this application, the aforementioned NM first pixels are the pixels other than M first pixels among the N first pixels.

[0075] Example 2, combined with Example 1, since point cloud point A corresponds to first pixel point A, point cloud point B corresponds to first pixel point B, and point cloud point C corresponds to first pixel point C, then first pixel point A, first pixel point B, and first pixel point C are the aforementioned M first pixel points, and first pixel point D and first pixel point E are the aforementioned NM first pixel points.

[0076] Thus, considering that the field of view (FOV) of dtof is limited, while binocular RGB often has a larger FOV, for RGB areas that are outside the coverage of dtof, this embodiment combines the first depth image with the first field of view image with a wider field of view to calculate the first disparity value, thereby effectively increasing the field of view (FOV) of the depth map and thus achieving the goal of increasing the depth map frame rate.

[0077] Step 203: The electronic device determines the second disparity information of each first pixel based on the first disparity value and the first disparity information corresponding to each first pixel.

[0078] In some embodiments of this application, the first disparity information is: the disparity information between the first pixel and its corresponding second pixel.

[0079] In some embodiments of this application, the second pixel is a pixel in the second field of view image.

[0080] In some embodiments of this application, the second field-of-view image and the first field-of-view image are images of different fields of view taken in the same scene.

[0081] In some embodiments of this application, the first disparity information is obtained by performing binocular stereo matching between the first field-of-view image and the second field-of-view image.

[0082] For example, binocular stereo matching refers to the operation of stereo matching of field-of-view images of the same target scene obtained from different field-of-view angles. The cost cube is a set of matching data obtained in binocular stereo matching, including each disparity value of each pixel and the cost value corresponding to each disparity value of each pixel.

[0083] In some embodiments of this application, the first disparity information includes a second disparity value between a first pixel and a second pixel, and a first generation value between the first pixel and the second pixel.

[0084] For example, the second disparity value mentioned above represents the offset of the second pixel relative to the first pixel.

[0085] For example, the first-generation value mentioned above represents the similarity between the first pixel and the second pixel.

[0086] The cost value is a numerical value that can measure the correlation between two pixels. Regardless of whether the two pixels are corresponding points in stereo matching, the matching cost value can be calculated through the matching cost function. The smaller the cost value, the greater the correlation, and the greater the probability that the two pixels are corresponding points.

[0087] In some embodiments of this application, the electronic device calculates a cost weight based on the first disparity value corresponding to each first pixel and the second disparity value in the first disparity information, and updates each first-generation value according to the cost weight to obtain the second disparity information.

[0088] In some embodiments of this application, the second disparity information includes the second disparity value of the first pixel and the second pixel, as well as the updated first-generation value between the first pixel and the second pixel.

[0089] Step 204: The electronic device determines the second depth value of each first pixel point according to each second parallax information, and generates a second depth map based on N second depth values.

[0090] Specifically, in some embodiments of this application, step 204 specifically includes steps 204a to 204c:

[0091] Step 204a: The electronic device determines the target generation value with the smallest generation value from the updated L first-generation values.

[0092] In some embodiments of this application, the above-mentioned L first cost values ​​are the cost values ​​between the first pixel and the corresponding L second pixels.

[0093] In some embodiments of this application, a target generation value with the smallest generation value is determined from the updated L first generation values, and this target generation value is used as the final generation value corresponding to the first pixel.

[0094] Step 204b: The electronic device determines the target disparity value from the L second disparity values ​​that corresponds to the same second pixel point as the target value.

[0095] In some embodiments of this application, since the first pixel corresponds to different second pixels at different disparity values, the electronic device takes the second pixel corresponding to the target disparity value as the target pixel. In other words, it indicates that the first pixel is at the target disparity value.

[0096] Step 204c: The electronic device calculates the second depth value based on the target parallax value.

[0097] In some embodiments of this application, the electronic device selects the smallest generation value based on the second generation value in the second field of view information, determines the second disparity value corresponding to the generation value as the optimal disparity value of the first pixel, and calculates the depth value of the first pixel using formula (2) based on the optimal disparity value. After obtaining the depth values ​​of all first pixels, the electronic device generates the final depth map.

[0098] For example, the above formula (2) is a variation of formula (1), and the specific formula is as follows:

[0099]

[0100] In the image processing method provided in this application embodiment, a first depth image is acquired, which includes M depth points, each depth point corresponding to a first depth value, where M is a positive integer; based on the M first depth values, N first disparity values ​​corresponding to N first pixels are determined, where the N first pixels are all pixels in the first field of view image, and N is a positive integer greater than M; based on the first disparity value and first disparity information corresponding to each first pixel, second disparity information for each first pixel is determined, where the first disparity information is the disparity information between the first pixel and its corresponding second pixel, and the second pixel is a pixel in the second field of view image, where the second field of view image and the first field of view image are images of different fields of view captured in the same scene; based on each second disparity information, a second depth value for each first pixel is determined, and a second depth map is generated based on the N second depth values. In this scheme, based on the depth value in a highly accurate depth image, N highly accurate disparity values ​​corresponding to N first pixels in the first field of view are determined. Then, these highly accurate disparity values ​​are fused and calculated with the first disparity information to correct each other, so as to obtain highly accurate second disparity information. The second depth value is determined based on the second disparity information, thereby improving the accuracy of the depth value. In this way, a depth map that is more consistent with the actual three-dimensional object can be generated, that is, the accuracy of the generated depth map is improved.

[0101] Furthermore, in some embodiments of this application, combined with Figure 1 ,like Figure 2 As shown, before step 203, "the electronic device determines the second disparity information of each first pixel based on the first disparity value and the first disparity information corresponding to each first pixel," the image processing method provided in this application embodiment further includes steps 301 to 303:

[0102] Step 301: The electronic device determines the L second pixels corresponding to the first pixel based on the first pixel and L second disparity values.

[0103] In some embodiments of this application, each of the first pixels corresponds to L second pixels, where L is a positive integer.

[0104] In some embodiments of this application, a first pixel is used to determine a second pixel based on a second disparity value, and so on, with each first pixel corresponding to L second pixels.

[0105] Example 3, combined with Example 1, shows that the first field of view image includes first pixel point A, first pixel point B, first pixel point C, first pixel point D, and first pixel point E. Taking first pixel point A as an example, when the second disparity value is 0, the pixel point with the same coordinates in the second field of view image is identified as second pixel point A1. When the second disparity value is 1, the pixel point with the same coordinates in the second field of view image shifted 1 pixel to the right is identified as second pixel point A2. When the second disparity value is 2, the pixel point with the same coordinates in the second field of view image shifted 2 pixels to the right is identified as second pixel point A3, and so on. Pixel points in the second field of view image under multiple second disparity values ​​are calculated and identified as multiple second pixel points corresponding to first pixel point A.

[0106] Step 302: The electronic device determines the first generation value of each first pixel based on the gray value of each first pixel and the gray values ​​of the L second pixels corresponding to the first pixel.

[0107] In some embodiments of this application, each of the first-generation values ​​described above is used to indicate the grayscale similarity between a first pixel and a second pixel.

[0108] Step 303: The electronic device determines L second disparity values ​​and L first generation values ​​as a first disparity information corresponding to the first pixel.

[0109] In some embodiments of this application, the electronic device performs stereo matching on the first field-of-view image and the second field-of-view image to obtain a cost cube, namely the aforementioned first disparity information.

[0110] In some embodiments of this application, the cost cube includes the cost value corresponding to each disparity value of each first pixel obtained by stereo matching.

[0111] For example, an electronic device can calculate the cost cube of the first field-of-view image using a cost calculation method.

[0112] For example, the cost calculation method described above includes any of the following: SAD (Sum of Absolute Differences) image matching algorithm and Census similarity algorithm. It should be noted that, in this application, other calculation methods besides those described above may also be used, and this application does not specifically limit their application.

[0113] For example, the process of calculating the cost cube is as follows: by calculating the cost value between the left image pixel (x, y) and the right image dmax pixels (i.e., (x, y) to (x+d, y)), a cost cube with length W, width H and height dmax can be obtained. The left image is the first field of view image and the right image is the second field of view image. The cost value can be denoted as C(x, y, d), where d is the disparity value between two pixels.

[0114] It is understood that the above cost calculation method is the same as steps 301 to 303 above.

[0115] Thus, for pixels without dTOF points, relatively accurate depth map information can be estimated using a binocular stereo matching algorithm, which can then be combined with the depth values ​​obtained by the dTOF sensor to generate a more accurate depth map.

[0116] Optionally, in some embodiments of this application, combined with Figure 2 ,like Figure 3 As shown, in conjunction with steps 301 to 303 above, step 203, "the electronic device determines a second disparity information based on the first disparity value and the first disparity information corresponding to each first pixel," specifically includes steps 203a to 203c:

[0117] Step 203a: The electronic device determines a cost weight based on the first disparity value corresponding to the first pixel and a second disparity value corresponding to each second pixel corresponding to the first pixel.

[0118] In some embodiments of this application, each of the above cost weights is used to characterize the error probability of a first-generation value corresponding to a second pixel.

[0119] For example, the error probability of the first generation value is used to indicate that the smaller the similarity between the second pixel corresponding to the first pixel and the first pixel, the lower the matching rate between them.

[0120] For example, the error probability of the first-generation value is proportional to the cost weight value; that is, the higher the error probability of the first-generation value, the higher the cost weight value.

[0121] In some embodiments of this application, the electronic device uses formula (3) to calculate the cost weight between each second disparity value and the corresponding first disparity value.

[0122] For example, the above formula (3) is:

[0123]

[0124] Taking the first pixel A as an example, x is a second disparity value of the first pixel A, and μ is the first disparity value of the first pixel A.

[0125] Step 203b: The electronic device updates a first-generation value corresponding to each second pixel point based on a cost weight corresponding to each second pixel point.

[0126] In some embodiments of this application, the electronic device uses formula (4) to weight and update each first-generation value.

[0127] For example, the above formula (4) is:

[0128]

[0129] Where C is the cost corresponding to pixel p. This indicates the updated value.

[0130] Step 203c: The electronic device determines L second disparity values ​​and the updated L first-generation values ​​as a second disparity information.

[0131] Since the cost weight represents the probability of an error occurring and its value is set to less than 1, when updating the cost value using the cost weight, the cost value can be further reduced, making it easier to select two corresponding pixels with high similarity, thereby obtaining a more accurate depth value.

[0132] Each of the above-described method embodiments, or various possible implementations of each method embodiment, can be executed individually or in combination of any two or more. The specific implementation can be determined according to actual usage requirements, and this application does not impose any restrictions on this.

[0133] The window blurring method provided in this application can be executed by an electronic device or a window blurring device. This application uses a window blurring device executing the window blurring method as an example to illustrate the window blurring device provided in this application.

[0134] Figure 4 A schematic diagram of a possible structure of the image processing apparatus involved in an embodiment of this application is shown. For example... Figure 4 As shown, the image processing device 700 may include: an acquisition module 701, a processing module 702, and a generation module 703.

[0135] The acquisition module 701 is used to acquire a first depth image, which includes M depth points, each depth point corresponding to a first depth value, where M is a positive integer. The processing module 702 is used to determine N first disparity values ​​corresponding to N first pixels based on the M first depth values ​​acquired by the acquisition module 701, where the N first pixels are all pixels in the first field of view image, and N is a positive integer greater than M. The processing module 702 is also used to determine second disparity information for each first pixel based on the first disparity value and first disparity information corresponding to each first pixel, where the first disparity information is the disparity information between the first pixel and its corresponding second pixel, and the second pixel is a pixel in the second field of view image, where the second field of view image and the first field of view image are images of different fields of view taken in the same scene. The processing module 702 is also used to determine the second depth value for each first pixel based on each second disparity information. The generation module 703 is used to generate a second depth map based on the N second depth values.

[0136] Optionally, in some embodiments of this application, the above-mentioned processing module 702 is specifically used for:

[0137] M depth points are projected onto the first field of view image to determine M first pixels, and the M first pixels correspond one-to-one with the M depth points;

[0138] Based on the first depth value of each depth point, determine the first disparity value of a first pixel corresponding to each depth point, and obtain M first disparity values.

[0139] The preset disparity values ​​are respectively determined as NM first disparity values ​​corresponding to NM first pixels, where NM first pixels are the pixels other than M first pixels among N first pixels;

[0140] The NM first disparity values ​​and the M first disparity values ​​are determined as N first disparity values.

[0141] Optionally, in some embodiments of this application, each of the first pixels corresponds to L second pixels, where L is a positive integer;

[0142] The aforementioned processing module 702 is further configured to, before determining the second disparity information of each first pixel based on the first disparity value and first disparity information corresponding to each first pixel:

[0143] Based on the first pixel and L second disparity values, determine the L second pixels corresponding to the first pixel;

[0144] Based on the gray value of each first pixel and the gray values ​​of the L second pixels corresponding to the first pixel, the first generation value of each first pixel is determined. Each first generation value is used to indicate the gray similarity between the first pixel and a second pixel.

[0145] The L second disparity values ​​and L first generation values ​​are used to determine a first disparity information corresponding to the first pixel.

[0146] Optionally, in some embodiments of this application, the above-mentioned processing module 702 is specifically used for:

[0147] Based on the first disparity value corresponding to the first pixel and a second disparity value corresponding to each second pixel, a cost weight is determined. Each cost weight is used to characterize the error probability of a first generation value corresponding to a second pixel.

[0148] Based on a cost weight corresponding to each second pixel, update a first-generation value corresponding to each second pixel;

[0149] The L second disparity values ​​and the updated L first-generation values ​​are used to determine a second disparity information.

[0150] Optionally, in some embodiments of this application, the above-mentioned processing module 702 is specifically used for:

[0151] From the updated L first-generation values, determine the target generation value with the minimum generation value;

[0152] From the L second disparity values, determine the target disparity value that corresponds to the same second pixel as the target cost value;

[0153] The second depth value is calculated based on the target disparity value.

[0154] In the image processing apparatus provided in this application embodiment, a first depth image is acquired, which includes M depth points, each depth point corresponding to a first depth value, where M is a positive integer; based on the M first depth values, N first disparity values ​​corresponding to N first pixels are determined, where the N first pixels are all pixels in the first field of view image, and N is a positive integer greater than M; based on the first disparity value and first disparity information corresponding to each first pixel, second disparity information for each first pixel is determined, where the first disparity information is the disparity information between the first pixel and its corresponding second pixel, and the second pixel is a pixel in the second field of view image, where the second field of view image and the first field of view image are images of different fields of view captured in the same scene; based on each second disparity information, a second depth value for each first pixel is determined, and a second depth map is generated based on the N second depth values. In this scheme, based on the depth value in a highly accurate depth image, N highly accurate disparity values ​​corresponding to N first pixels in the first field of view are determined. Then, these highly accurate disparity values ​​are fused and calculated with the first disparity information to correct each other, so as to obtain highly accurate second disparity information. The second depth value is determined based on the second disparity information, thereby improving the accuracy of the depth value. In this way, a depth map that is more consistent with the actual three-dimensional object can be generated, that is, the accuracy of the generated depth map is improved.

[0155] The image processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0156] The image processing device in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0157] The image processing apparatus provided in this application embodiment can implement the various processes implemented in the image processing method embodiment, and will not be described again here to avoid repetition.

[0158] Optionally, such as Figure 5 As shown, this application embodiment also provides an electronic device 800, including a processor 801 and a memory 802. The memory 802 stores a program or instructions that can run on the processor 801. When the program or instructions are executed by the processor 801, they implement the various steps of the above-described image processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0159] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0160] Figure 6 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0161] The electronic device 100 includes, but is not limited to, components such as: radio frequency unit 101, network module 102, audio output unit 103, input unit 104, sensor 105, display unit 106, user input unit 107, interface unit 108, memory 109, and processor 110.

[0162] Those skilled in the art will understand that the electronic device 100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 6 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0163] The processor 110 is configured to acquire a first depth image, which includes M depth points, each depth point corresponding to a first depth value, where M is a positive integer; the processor 110 is configured to determine N first disparity values ​​corresponding to N first pixels based on the M first depth values, where the N first pixels are all pixels in the first field of view image, and N is a positive integer greater than M; the processor 110 is further configured to determine second disparity information for each first pixel based on the first disparity value and first disparity information corresponding to each first pixel, where the first disparity information is the disparity information between the first pixel and its corresponding second pixel, and the second pixel is a pixel in the second field of view image, where the second field of view image and the first field of view image are images of different fields of view captured in the same scene; the processor 110 is further configured to determine the second depth value for each first pixel based on each second disparity information; and the processor 110 is configured to generate a second depth map based on the N second depth values.

[0164] Optionally, in some embodiments of this application, the processor 110 is specifically used for:

[0165] M depth points are projected onto the first field of view image to determine M first pixels, and the M first pixels correspond one-to-one with the M depth points;

[0166] Based on the first depth value of each depth point, determine the first disparity value of a first pixel corresponding to each depth point, and obtain M first disparity values.

[0167] The preset disparity values ​​are respectively determined as NM first disparity values ​​corresponding to NM first pixels, where NM first pixels are the pixels other than M first pixels among N first pixels;

[0168] The NM first disparity values ​​and the M first disparity values ​​are determined as N first disparity values.

[0169] Optionally, in some embodiments of this application, each of the first pixels corresponds to L second pixels, where L is a positive integer;

[0170] The processor 110 described above is further configured to, before determining the second disparity information of each first pixel based on the first disparity value and the first disparity information corresponding to each first pixel:

[0171] Based on the first pixel and L second disparity values, determine the L second pixels corresponding to the first pixel;

[0172] Based on the gray value of each first pixel and the gray values ​​of the L second pixels corresponding to the first pixel, the first generation value of each first pixel is determined. Each first generation value is used to indicate the gray similarity between the first pixel and a second pixel.

[0173] The L second disparity values ​​and L first generation values ​​are used to determine a first disparity information corresponding to the first pixel.

[0174] Optionally, in some embodiments of this application, the above-mentioned processing module 702 is specifically used for:

[0175] Based on the first disparity value corresponding to the first pixel and a second disparity value corresponding to each second pixel, a cost weight is determined. Each cost weight is used to characterize the error probability of a first generation value corresponding to a second pixel.

[0176] Based on a cost weight corresponding to each second pixel, update a first-generation value corresponding to each second pixel;

[0177] The L second disparity values ​​and the updated L first-generation values ​​are used to determine a second disparity information.

[0178] Optionally, in some embodiments of this application, the processor 110 is specifically used for:

[0179] From the updated L first-generation values, determine the target generation value with the minimum generation value;

[0180] From the L second disparity values, determine the target disparity value that corresponds to the same second pixel as the target cost value;

[0181] The second depth value is calculated based on the target disparity value.

[0182] In the electronic device provided in this application embodiment, a first depth image is acquired, which includes M depth points, each depth point corresponding to a first depth value, where M is a positive integer; based on the M first depth values, N first disparity values ​​corresponding to N first pixels are determined, where the N first pixels are all pixels in the first field of view image, and N is a positive integer greater than M; based on the first disparity value and first disparity information corresponding to each first pixel, second disparity information for each first pixel is determined, where the first disparity information is the disparity information between the first pixel and its corresponding second pixel, and the second pixel is a pixel in the second field of view image, where the second field of view image and the first field of view image are images of different fields of view captured in the same scene; based on each second disparity information, a second depth value for each first pixel is determined, and a second depth map is generated based on the N second depth values. In this scheme, based on the depth value in a highly accurate depth image, N highly accurate disparity values ​​corresponding to N first pixels in the first field of view are determined. Then, these highly accurate disparity values ​​are fused and calculated with the first disparity information to correct each other, so as to obtain highly accurate second disparity information. The second depth value is determined based on the second disparity information, thereby improving the accuracy of the depth value. In this way, a depth map that is more consistent with the actual three-dimensional object can be generated, that is, the accuracy of the generated depth map is improved.

[0183] It should be understood that, in this embodiment, the input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042. The GPU 1041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 107 includes at least one of a touch panel 1071 and other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0184] The memory 109 can be used to store software programs and various data. The memory 109 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 109 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 109 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0185] Processor 110 may include one or more processing units; optionally, processor 110 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 110.

[0186] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described image processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0187] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0188] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described image processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0189] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0190] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described image processing method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0191] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0192] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0193] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image processing method, characterized in that, The method includes: Obtain a first depth image, which includes M depth points, each depth point corresponding to a first depth value, where M is a positive integer; Based on the M first depth values, determine the N first disparity values ​​corresponding to the N first pixels, where the N first pixels are all pixels in the first field of view image, and N is a positive integer greater than M; Based on the first disparity value and first disparity information corresponding to each first pixel, the second disparity information of each first pixel is determined respectively. The first disparity information is the disparity information between the first pixel and its corresponding second pixel. The second pixel is a pixel in the second field of view image. The second field of view image and the first field of view image are images of different fields of view taken in the same scene. Based on each of the second disparity information, the second depth value of each first pixel is determined, and a second depth map is generated based on the N second depth values; The step of determining N first disparity values ​​corresponding to N first pixels based on M first depth values ​​includes: The M depth points are projected onto the first field of view image to determine the M first pixels, and the M first pixels correspond one-to-one with the M depth points; Based on the first depth value of each depth point, the first disparity value of the first pixel corresponding to each depth point is determined to obtain M first disparity values. The preset disparity values ​​are respectively determined as NM first disparity values ​​corresponding to NM first pixels, where NM first pixels are the pixels other than M first pixels among N first pixels; The NM first disparity values ​​and the M first disparity values ​​are determined to be N first disparity values.

2. The method according to claim 1, characterized in that, Each of the first pixels corresponds to L second pixels, where L is a positive integer; Before determining the second disparity information of each first pixel based on the first disparity value and first disparity information corresponding to each first pixel, the method further includes: Based on the first pixel and L second disparity values, determine the L second pixels corresponding to the first pixel; Based on the gray value of each first pixel and the gray values ​​of L second pixels corresponding to the first pixel, the first generation value of each first pixel is determined. Each first generation value is used to indicate the gray similarity between the first pixel and a second pixel. The L second disparity values ​​and L first generation values ​​are used to determine one first disparity information corresponding to the first pixel.

3. The method according to claim 2, characterized in that, The step of determining the second disparity information for each first pixel based on the first disparity value and first disparity information corresponding to each first pixel includes: Based on the first disparity value corresponding to the first pixel and a second disparity value corresponding to each second pixel corresponding to the first pixel, a cost weight is determined, and each cost weight is used to characterize the error probability of a first generation value corresponding to a second pixel. Based on the cost weight corresponding to each second pixel, the first generation value corresponding to each second pixel is updated; The L second disparity values ​​and the updated L first generation values ​​are used to determine a second disparity information.

4. The method according to claim 3, characterized in that, The step of determining the second depth value of each first pixel based on each of the second disparity information includes: From the updated L first-generation values, determine the target generation value with the smallest generation value; From the L second disparity values, determine the target disparity value that corresponds to the same second pixel point as the target cost value; The second depth value is calculated based on the target disparity value.

5. An image processing apparatus, characterized in that, The image processing device includes: an acquisition module, a processing module, and a generation module; The acquisition module is used to acquire a first depth image, which includes M depth points, one depth point corresponding to a first depth value, where M is a positive integer; The processing module is used to determine N first disparity values ​​corresponding to N first pixels based on M first depth values ​​obtained by the acquisition module, wherein the N first pixels are all pixels in the first field of view image, and N is a positive integer greater than M. The processing module is further configured to determine the second disparity information of each first pixel based on the first disparity value and the first disparity information corresponding to each first pixel, wherein the first disparity information is the disparity information between the first pixel and its corresponding second pixel, the second pixel is a pixel in the second field of view image, and the second field of view image and the first field of view image are images of different fields of view taken in the same scene. The processing module is further configured to determine the second depth value of each first pixel based on each of the second disparity information. The generation module is used to generate a second depth map based on N second depth values; Specifically, the processing module is used for: The M depth points are projected onto the first field of view image to determine the M first pixels, and the M first pixels correspond one-to-one with the M depth points; Based on the first depth value of each depth point, the first disparity value of the first pixel corresponding to each depth point is determined to obtain M first disparity values. The preset disparity values ​​are respectively determined as NM first disparity values ​​corresponding to NM first pixels, where NM first pixels are the pixels other than M first pixels among N first pixels; The NM first disparity values ​​and the M first disparity values ​​are determined to be N first disparity values.

6. The apparatus according to claim 5, characterized in that, Each of the first pixels corresponds to L second pixels, where L is a positive integer; The processing module is further configured to, before determining the second disparity information of each first pixel based on the first disparity value and the first disparity information corresponding to each first pixel: Based on the first pixel and L second disparity values, determine the L second pixels corresponding to the first pixel; Based on the gray value of each first pixel and the gray values ​​of the L second pixels corresponding to the first pixel, the first generation value of each first pixel is determined. Each first generation value is used to indicate the gray similarity between the first pixel and a second pixel. The L second disparity values ​​and L first generation values ​​are used to determine one first disparity information corresponding to the first pixel.

7. The apparatus according to claim 6, characterized in that, The processing module is specifically used for: Based on the first disparity value corresponding to the first pixel and a second disparity value corresponding to each second pixel corresponding to the first pixel, a cost weight is determined, and each cost weight is used to characterize the error probability of a first generation value corresponding to a second pixel. Based on the cost weight corresponding to each second pixel, the first generation value corresponding to each second pixel is updated; The L second disparity values ​​and the updated L first generation values ​​are used to determine a second disparity information.

8. The apparatus according to claim 7, characterized in that, The processing module is specifically used for: From the updated L first-generation values, determine the target generation value with the smallest generation value; From the L second disparity values, determine the target disparity value that corresponds to the same second pixel point as the target cost value; The second depth value is calculated based on the target disparity value.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the image processing method as described in any one of claims 1 to 4.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the image processing method as described in any one of claims 1 to 4.

Images