Methods for determining image parallax, electronic devices, and storage media.

Abstract

This application relates to a method, electronic device, and storage medium for determining image disparity. The method involves constructing a first pyramid by downsampling a first image N times according to a sampling rule, and constructing a second pyramid by downsampling a second image N times according to the same sampling rule. Stereo matching calculations are then performed on the Nth layer image of the first pyramid and the Nth layer image of the second pyramid to obtain a disparity map of the Nth layer image of the first pyramid. Cost calculations are then performed on the 0th layer image of the first pyramid and the 0th layer image of the second pyramid to obtain multiple cost values ​​corresponding to each pixel in the 0th layer image of the first pyramid. Finally, based on the disparity map of the Nth layer image of the first pyramid and the multiple cost values ​​corresponding to each pixel in the 0th layer image of the first pyramid, a disparity map of the 0th layer image of the first pyramid is obtained. This method significantly improves the efficiency of obtaining a disparity map by performing stereo matching calculations on the first image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method for determining image parallax, an electronic device, and a storage medium. Background Technology

[0002] Stereo matching has always been a research hotspot in binocular vision. Binocular cameras capture two viewpoint images of the same scene from the left and right, and stereo matching algorithms are used to obtain disparity maps.

[0003] Currently, methods for achieving stereo matching mainly include: stereo calibration of the images from the left and right cameras, followed by cost calculation, cost aggregation, disparity estimation, and disparity optimization to obtain a disparity map. However, if the pixel size of the left and right viewpoint images is large, the above-mentioned stereo matching method becomes very time-consuming. Alternatively, if the frequency of acquiring the left and right viewpoint images is high, the efficiency of the above-mentioned stereo matching method is low and cannot meet the requirements for high efficiency. Therefore, this method has become a key bottleneck limiting the improvement of the frame rate of stereo matching. Summary of the Invention

[0004] Therefore, it is necessary to provide a method for determining image parallax, an electronic device, and a storage medium to address the aforementioned technical problems.

[0005] In a first aspect, one embodiment of this application provides a method for determining image parallax, the method comprising:

[0006] A first pyramid is constructed by downsampling the first image N times according to the sampling rules, and a second pyramid is constructed by downsampling the second image N times according to the sampling rules; wherein, N is an integer greater than or equal to 1, the top layer of the first pyramid is the Nth layer, the bottom layer of the first pyramid is the 0th layer, the resolution of the images from the 0th layer to the Nth layer in the first pyramid gradually decreases, and the image from the 0th layer in the first pyramid is the first image; the top layer of the second pyramid is the Nth layer, the bottom layer of the second pyramid is the 0th layer, the resolution of the images from the 0th layer to the Nth layer in the second pyramid gradually decreases, and the image from the 0th layer in the second pyramid is the second image;

[0007] Stereo matching calculations are performed on the Nth layer image of the first pyramid and the Nth layer image of the second pyramid to obtain the disparity map of the Nth layer image of the first pyramid;

[0008] Cost calculations are performed on the images at layer 0 of the first pyramid and the images at layer 0 of the second pyramid to obtain multiple cost values ​​corresponding to each pixel in the image at layer 0 of the first pyramid.

[0009] Based on the disparity map of the Nth layer image in the first pyramid and the multiple cost values ​​corresponding to each pixel in the 0th layer image in the first pyramid, the disparity map of the 0th layer image in the first pyramid is obtained.

[0010] In one embodiment, obtaining the disparity map of the image at layer N of the first pyramid based on the disparity map of the image at layer N and the multiple cost values ​​corresponding to each pixel in the image at layer 0 of the first pyramid includes:

[0011] Cost calculations are performed on the (N-1)th layer image of the first pyramid and the (N-1)th layer image of the second pyramid to obtain multiple cost values ​​corresponding to each pixel in the (N-1)th layer image of the first pyramid;

[0012] Based on the disparity map of the Nth layer image in the first pyramid and the multiple cost values ​​corresponding to each pixel in the (N-1)th layer image in the first pyramid, the disparity map of the (N-1)th layer image in the first pyramid is determined.

[0013] Based on the disparity map of the (N-1)th layer image in the first pyramid and the multiple cost values ​​corresponding to each pixel in the 0th layer image in the first pyramid, the disparity map of the 0th layer image in the first pyramid is obtained.

[0014] In one embodiment, constructing the first pyramid by downsampling the first image N times according to the sampling rules includes:

[0015] The first image is downsampled N times according to the sampling rules to obtain N layers of third images with different resolutions;

[0016] The first image and the N layers of third images with different resolutions are combined in descending order of resolution to construct the first pyramid;

[0017] The step of downsampling the second image N times according to the sampling rule to construct the second pyramid includes:

[0018] The second image is downsampled N times according to the sampling rule to obtain N layers of fourth images with different resolutions;

[0019] The second image and the N layers of fourth images with different resolutions are combined in descending order of resolution to construct the second pyramid.

[0020] In one embodiment, the step of performing stereo matching calculations on the Nth layer image of the first pyramid and the Nth layer image of the second pyramid to obtain a disparity map of the Nth layer image of the first pyramid includes:

[0021] Cost calculations are performed on the Nth layer image of the first pyramid and the Nth layer image of the second pyramid to obtain multiple cost values ​​corresponding to each pixel in the Nth layer image of the first pyramid;

[0022] The minimum value among the multiple values ​​corresponding to each pixel in the Nth layer of the first pyramid is taken as the target value corresponding to the pixel.

[0023] Based on the target cost value corresponding to each pixel in the Nth layer image of the first pyramid, the disparity value corresponding to each pixel in the Nth layer image of the first pyramid is obtained, so as to obtain the disparity map of the Nth layer image of the first pyramid.

[0024] In one embodiment, determining the disparity map of the (N-1)th layer image in the first pyramid based on the disparity map of the Nth layer image in the first pyramid and multiple cost values ​​corresponding to each pixel in the (N-1)th layer image in the first pyramid includes:

[0025] Based on the disparity map of the Nth layer image in the first pyramid and the first preset value, determine the range of comparative disparity values ​​for each pixel in the (N-1)th layer image of the first pyramid;

[0026] Multiple first target disparity values ​​are obtained for each pixel in the (N-1)th layer of the first pyramid image based on multiple cost values.

[0027] The first target disparity value among the multiple first target disparity values ​​corresponding to each pixel in the N-1th layer image of the first pyramid, which falls within the range of the comparison disparity values ​​of the corresponding pixel in the N-1th layer image of the first pyramid, is taken as the target disparity value of the corresponding pixel in the N-1th layer image of the first pyramid.

[0028] Based on the target disparity value of the corresponding pixel in the (N-1)th layer image of the first pyramid, determine the disparity map of the (N-1)th layer image of the first pyramid.

[0029] In one embodiment, determining the disparity map of the (N-1)th layer image in the first pyramid based on the target disparity value of the corresponding pixel in the (N-1)th layer image of the first pyramid includes:

[0030] When any pixel in the (N-1)th layer of the first pyramid has two or more target disparity values, the minimum target disparity value among the target disparity values ​​corresponding to any pixel in the (N-1)th layer of the first pyramid shall be taken as the disparity value of the corresponding pixel in the (N-1)th layer of the first pyramid.

[0031] When any pixel in the (N-1)th layer of the first pyramid has a target disparity value, the target disparity value corresponding to any pixel in the (N-1)th layer of the first pyramid is taken as the disparity value of the corresponding pixel in the (N-1)th layer of the first pyramid.

[0032] In one embodiment, determining the range of comparative disparity values ​​for each pixel in the (N-1)th layer of the first pyramid based on the disparity map of the Nth layer image in the first pyramid includes:

[0033] According to the sampling rule, target pixels corresponding to each pixel in the (N-1)th layer of the first pyramid are determined in the Nth layer image of the first pyramid;

[0034] Based on the disparity map of the Nth layer image in the first pyramid, the disparity value corresponding to the target pixel in the Nth layer image in the first pyramid is used as the comparison disparity value of the corresponding pixel in the (N-1)th layer image in the first pyramid.

[0035] Based on the comparison disparity value of each pixel in the (N-1)th layer image of the first pyramid and the first preset value, the range of comparison disparity values ​​of each pixel in the (N-1)th layer image of the first pyramid is determined.

[0036] In one embodiment, the step of performing stereo matching calculations on the Nth layer image of the first pyramid and the Nth layer image of the second pyramid to obtain a disparity map of the Nth layer image of the first pyramid includes:

[0037] The SSBM or AD-Census algorithm is used to perform stereo matching calculations on the Nth layer image in the first pyramid and the Nth layer image in the second pyramid to obtain the disparity map of the Nth layer image in the first pyramid.

[0038] In a second aspect, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0039] Thirdly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in the first aspect.

[0040] The image disparity determination method, electronic device, and storage medium described in the above embodiments construct a first pyramid by downsampling a first image N times according to sampling rules, and construct a second pyramid by downsampling a second image N times according to sampling rules. Stereo matching calculations are then performed on the Nth layer image of the first pyramid and the Nth layer image of the second pyramid to obtain a disparity map of the Nth layer image of the first pyramid. Cost calculations are then performed on the 0th layer image of the first pyramid and the 0th layer image of the second pyramid to obtain multiple cost values ​​corresponding to each pixel in the 0th layer image of the first pyramid. Finally, a disparity map of the 0th layer image of the first pyramid is obtained based on the disparity map of the Nth layer image of the first pyramid and the multiple cost values ​​corresponding to each pixel in the 0th layer image of the first pyramid. In this method, complete stereo matching calculations are performed only on the Nth layer image of the first pyramid. For other layers, cost calculations are performed only based on the disparity map of the upper layer image, without requiring time-consuming cost aggregation and disparity calculations, greatly improving the efficiency of obtaining a disparity map from the first image through stereo matching calculations. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 A schematic diagram of the structure of an electronic device provided in one embodiment;

[0043] Figure 2 A flowchart illustrating the steps of an image disparity determination method provided in one embodiment;

[0044] Figure 3 for Figure 2 A flowchart illustrating one implementation of S104 in the embodiment;

[0045] Figure 4 for Figure 3 A flowchart illustrating one implementation of S202 in the embodiment;

[0046] Figure 5 A schematic diagram of a parallax range provided for one embodiment;

[0047] Figure 6 A schematic diagram of a pyramid structure provided for one embodiment;

[0048] Figure 7 for Figure 4A flowchart illustrating one implementation of S301 in the embodiment;

[0049] Figure 8 for Figure 2 A flowchart illustrating one implementation of S101 in the embodiment;

[0050] Figure 9 A schematic diagram of a pyramid structure is provided for one embodiment;

[0051] Figure 10 for Figure 2 A flowchart illustrating another implementation of S101 in the embodiment;

[0052] Figure 11 for Figure 2 A flowchart illustrating one implementation of S102 in the embodiment;

[0053] Figure 12 A schematic diagram of the structure of an image parallax determination device provided in one embodiment. Detailed Implementation

[0054] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the specific embodiments of this application are described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0055] The technical solution of this application and how it solves the technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0056] The serial numbers assigned to components in this document, such as "first" and "second," are used solely to distinguish the described objects and have no sequential or technical meaning. The terms "connection" and "linkage" used in this application, unless otherwise specified, include both direct and indirect connections (linkages). It should be understood that the terms "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are used solely for the convenience of describing this application and simplifying the description. They do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0057] In this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0058] The image disparity determination method provided in this application embodiment can be applied to, for example, Figure 1 The electronic device shown is a terminal, and its internal structure diagram can be as follows. Figure 1 As shown, the electronic device includes a processor, memory, communication interface, display screen, and input device. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a method for determining image parallax. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0059] Those skilled in the art will understand that Figure 1The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0060] In one embodiment, such as Figure 2 As shown, a method for determining image parallax is provided, which can be applied to... Figure 1 Taking an electronic device as an example, the explanation includes the following steps:

[0061] S101, construct a first pyramid by downsampling the first image N times according to the sampling rules, and construct a second pyramid by downsampling the second image N times according to the sampling rules.

[0062] Where N is an integer greater than or equal to 1; the top layer of the first pyramid is layer N, followed by layers N-1, N-2, and so on, until the bottom layer is layer 0. The resolution of the images from layer 0 to layer N in the first pyramid gradually decreases, and the resolution of the images from layer 0 to layer N in the second pyramid also gradually decreases. Optionally, the first image is a corrected left camera image, i.e., after acquiring the left camera image, the left camera image is corrected to obtain the first image; correspondingly, the second image is a corrected right camera image, i.e., after acquiring the right camera image, the right camera image is corrected to obtain the second image. Optionally, the first image can also be a corrected right camera image, i.e., after acquiring the right camera image, the right camera image is corrected to obtain the first image; and the second image can be a corrected left camera image, i.e., after acquiring the left camera image, the left camera image is corrected to obtain the second image.

[0063] In this embodiment, the electronic device can pre-connect a left camera and a right camera, or directly connect a stereo camera, and obtain images from the left camera and the right camera. Then, it corrects the images obtained from the left and right cameras to obtain corrected left and right camera images. Alternatively, the electronic device directly obtains left and right camera images from the stereo camera, and then corrects the obtained left and right camera images to obtain a first image and a second image. Based on the above method, when the electronic device acquires the first and second images, it can select a sampling rule to downsample the first image N times to construct a first pyramid corresponding to the first image, and select a sampling rule to downsample the second image N times to construct a second pyramid corresponding to the second image. It should be noted that the sampling rule used for downsampling the first image is the same as the sampling rule used for downsampling the second image, that is, the number of layers in the constructed first and second pyramids is the same, and the resolution of each layer is consistent.

[0064] S102, Perform stereo matching calculation on the Nth layer image in the first pyramid and the Nth layer image in the second pyramid to obtain the disparity map of the Nth layer image in the first pyramid.

[0065] In this system, the Nth layer image in the first pyramid is the image with the lowest resolution among all layers in the first pyramid. Stereo matching calculation includes cost calculation; optionally, stereo matching calculation may also include cost calculation, cost aggregation, disparity estimation, and disparity optimization. After the electronic device constructs the first and second pyramids, it can first perform cost calculation on the Nth layer image in the first pyramid and the Nth layer image in the second pyramid to complete the stereo matching calculation, obtaining the disparity map of the Nth layer image in the first pyramid, and correspondingly obtaining the disparity map of the Nth layer image in the second pyramid. Optionally, after the electronic device constructs the first and second pyramids, it can also sequentially perform cost calculation, cost aggregation, disparity estimation, and disparity optimization on the Nth layer image in the first pyramid and the Nth layer image in the second pyramid to complete the full stereo matching calculation, obtaining the disparity map of the Nth layer image in the first pyramid, and correspondingly obtaining the disparity map of the Nth layer image in the second pyramid.

[0066] S103, calculate the cost of the image at layer 0 in the first pyramid and the image at layer 0 in the second pyramid to obtain multiple cost values ​​corresponding to each pixel in the image at layer 0 in the first pyramid.

[0067] In this embodiment, a corresponding cost calculation method can be used to calculate the cost of the image at layer 0 in the first pyramid and the image at layer 0 in the second pyramid, so as to obtain multiple cost values ​​corresponding to each pixel in the image at layer 0 in the first pyramid; correspondingly, multiple cost values ​​corresponding to each pixel in the image at layer 0 in the second pyramid can also be obtained.

[0068] Accordingly, using the method described in S103, the electronic device can also perform cost calculations on the image of layer 0 in the first pyramid and the image of layer 0 in the second pyramid to obtain multiple cost values ​​corresponding to each pixel in the image of layer 0 in the second pyramid.

[0069] S104. Based on the disparity map of the Nth layer image in the first pyramid and the multiple cost values ​​corresponding to each pixel in the 0th layer image in the first pyramid, obtain the disparity map of the 0th layer image in the first pyramid.

[0070] This embodiment relates to a method for determining the disparity map of a lower-level image based on the disparity map of the upper-level image and the cost value of the lower-level image in the first pyramid, thereby obtaining the disparity map of the 0th-level image in the first pyramid. Specifically, this can be implemented in the following three ways:

[0071] The first implementation method is as follows: when the electronic device obtains the disparity map of the Nth layer image in the first pyramid, that is, when it obtains the disparity value of each pixel in the disparity map of the Nth layer image, it can further use the disparity value of each pixel in the Nth layer image as the prior disparity value, and then combine it with the multiple cost values ​​of each pixel in the 0th layer image to determine the disparity value of each pixel in the 0th layer image, thereby obtaining the disparity map of the 0th layer image in the first pyramid. Finally, the disparity map of the 0th layer image in the first pyramid can be determined as the disparity map corresponding to the first image.

[0072] The second implementation involves the electronic device acquiring the disparity map of the Nth layer image in the first pyramid, i.e., acquiring the disparity value of each pixel in the disparity map of the Nth layer image. This disparity value can then be used as a priori disparity value, combined with multiple cost values ​​corresponding to each pixel in the (N-1)th layer image, to determine the disparity value of each pixel in the (N-1)th layer image, thus obtaining the disparity map of the (N-1)th layer image in the first pyramid. The same method is then used to sequentially obtain the disparity maps of the (N-2), (N-3), ..., (NM)th layers in the first pyramid, where M is an integer less than N. Finally, the disparity value of each pixel in the (NM)th layer image is used as a priori disparity value, combined with multiple cost values ​​of each pixel in the 0th layer image, to determine the disparity value of each pixel in the 0th layer image, thus obtaining the disparity map of the 0th layer image in the first pyramid. Finally, the disparity map of the 0th layer image in the first pyramid can be determined as the disparity map corresponding to the first image.

[0073] The third implementation method is as follows: when the electronic device obtains the disparity map of the Nth layer image in the first pyramid, that is, when it obtains the disparity value of each pixel in the disparity map of the Nth layer image, it can further use the disparity value of each pixel in the Nth layer image as the prior disparity value, and then combine it with the multiple cost values ​​corresponding to each pixel in the (N-1)th layer image to determine the disparity value of each pixel in the (N-1)th layer image, thereby obtaining the disparity map of the (N-1)th layer image in the first pyramid. Then, the same method is used to obtain the disparity maps of the (N-2)th layer image, the (N-3)th layer image, ..., the 0th layer image in the first pyramid in turn. Finally, the disparity map of the 0th layer image in the first pyramid is determined as the disparity map corresponding to the first image.

[0074] Accordingly, using the method described in S104, the electronic device can also obtain the disparity map of the image of the 0th layer in the second pyramid based on the disparity map of the Nth layer image in the second pyramid and the multiple cost values ​​corresponding to each pixel in the 0th layer image in the second pyramid.

[0075] The image disparity determination method provided in the above embodiments constructs a first pyramid by downsampling the first image N times according to sampling rules, and constructs a second pyramid by downsampling the second image N times according to sampling rules. Stereo matching calculations are then performed on the Nth layer image of the first pyramid and the Nth layer image of the second pyramid to obtain a disparity map of the Nth layer image of the first pyramid. Cost calculations are then performed on the 0th layer image of the first pyramid and the 0th layer image of the second pyramid to obtain multiple cost values ​​corresponding to each pixel in the 0th layer image of the first pyramid. Finally, a disparity map of the 0th layer image of the first pyramid is obtained based on the disparity map of the Nth layer image of the first pyramid and the multiple cost values ​​corresponding to each pixel in the 0th layer image of the first pyramid. In this method, complete stereo matching calculations are performed only on the Nth layer image of the first pyramid. For other layers, cost calculations are performed only based on the disparity map of the upper layer image, without requiring time-consuming cost aggregation and disparity calculations, greatly improving the efficiency of obtaining a disparity map from the first image through stereo matching calculations.

[0076] In one embodiment, a specific implementation of S104 is provided, namely, a method for iteratively calculating the disparity map of the lowest layer image in the pyramid based on the disparity map of the lower-resolution image and the cost value of the higher-resolution image in every two adjacent layers of the pyramid. Figure 3 As shown, this implementation method includes:

[0077] S201, calculate the cost of the (N-1)th layer image in the first pyramid and the (N-1)th layer image in the second pyramid to obtain multiple cost values ​​corresponding to each pixel in the (N-1)th layer image in the first pyramid.

[0078] In this embodiment, a corresponding cost calculation method can be used to calculate the cost of the (N-1)th layer image in the first pyramid and the (N-1)th layer image in the second pyramid, so as to obtain multiple cost values ​​corresponding to each pixel in the (N-1)th layer image in the first pyramid; correspondingly, multiple cost values ​​corresponding to each pixel in the (N-1)th layer image in the second pyramid can also be obtained.

[0079] S202, based on the disparity map of the Nth layer image in the first pyramid and the multiple cost values ​​corresponding to each pixel in the (N-1)th layer image in the first pyramid, determine the disparity map of the (N-1)th layer image in the first pyramid.

[0080] When the electronic device obtains the disparity map of the Nth layer image in the first pyramid, that is, when it obtains the disparity value of each pixel in the disparity map of the Nth layer image, it can further use the disparity value of each pixel in the Nth layer image as a priori disparity value, and then combine it with the multiple cost values ​​of each pixel in the (N-1)th layer image to determine the disparity value of each pixel in the (N-1)th layer image, thereby obtaining the disparity map of the (N-1)th layer image in the first pyramid.

[0081] S203, based on the disparity map of the N-1th layer image in the first pyramid and the multiple cost values ​​corresponding to each pixel in the 0th layer image in the first pyramid, obtain the disparity map of the 0th layer image in the first pyramid.

[0082] This embodiment can be implemented in three ways, namely:

[0083] The first implementation method is as follows: When the electronic device obtains the disparity map of the N-1th layer image in the first pyramid, that is, when it obtains the disparity value of each pixel in the disparity map of the N-1th layer image, it can further use the disparity value of each pixel in the N-1th layer image as the prior disparity value, and then combine it with the multiple cost values ​​of the corresponding pixel in the 0th layer image to determine the disparity value of each pixel in the 0th layer image, thereby obtaining the disparity map of the 0th layer image in the first pyramid.

[0084] The second implementation method is as follows: When the electronic device obtains the disparity map of the N-1th layer image in the first pyramid, that is, when it obtains the disparity value of each pixel in the disparity map of the N-1th layer image, it can further use the disparity value of each pixel in the N-1th layer image as a priori disparity value, and then combine it with the multiple cost values ​​corresponding to each pixel in the N-2th layer image to determine the disparity value of each pixel in the N-2th layer image, thereby obtaining the disparity map of the N-2th layer image in the first pyramid. Then, the same method is used to obtain the disparity maps of the N-3th layer image, the N-4th layer image, ..., the NMth layer image in the first pyramid in sequence, where M is an integer less than N. Finally, the disparity value of each pixel in the NMth layer image is used as a priori disparity value, and then combined with the multiple cost values ​​of each pixel in the 0th layer image to determine the disparity value of each pixel in the 0th layer image, thereby obtaining the disparity map of the 0th layer image in the first pyramid.

[0085] The third implementation method is as follows: When the electronic device obtains the disparity map of the N-1th layer image in the first pyramid, that is, when it obtains the disparity value of each pixel in the disparity map of the N-1th layer image, it can further use the disparity value of each pixel in the N-1th layer image as the prior disparity value, and then combine it with the multiple cost values ​​of the corresponding pixel in the N-2th layer image to determine the disparity value of each pixel in the N-2th layer image, thereby obtaining the disparity map of the N-2th layer image in the first pyramid. Then, the same method is used to obtain the disparity maps of the N-3th layer image, the N-4th layer image, ... the 1st layer image in the first pyramid, until the disparity map of the 0th layer image in the first pyramid is obtained.

[0086] Furthermore, a specific implementation of the above S202 is provided, such as... Figure 4 As shown, the above-mentioned S202 "determines the disparity map of the (N-1)th layer image in the first pyramid based on the disparity map of the Nth layer image in the first pyramid and the multiple cost values ​​corresponding to each pixel in the (N-1)th layer image in the first pyramid" includes:

[0087] S301, based on the disparity map of the Nth layer image in the first pyramid and the first preset value, determine the range of comparison disparity values ​​for each pixel in the (N-1)th layer image in the first pyramid.

[0088] When the electronic device acquires the disparity map of the Nth layer image in the first pyramid, it can determine the disparity value of each pixel in the Nth layer image. The electronic device can further determine a disparity range for each pixel based on its disparity value. For example, taking a single pixel as an example, the disparity value of that pixel can be increased or decreased to obtain a numerical range containing that disparity value. This numerical range can then be determined as the disparity range for that pixel. Figure 5As shown, d0 is the disparity value of a single pixel. Increasing or decreasing d0 by 5 forms a disparity range of D{d0-5, 0+5}. The method for determining the disparity range of other pixels is the same as described above and will not be repeated here. When the disparity range corresponding to each pixel in the Nth layer of the first pyramid is determined using the above method, the disparity range corresponding to each pixel in the Nth layer of the first pyramid can be directly used as the range of comparison disparity values ​​corresponding to the corresponding pixels in the (N-1)th layer of the first pyramid. It should be noted that because the resolution of the Nth layer image in the first pyramid is lower than that of the (N-1)th layer image, the number of pixels contained in the Nth layer image is less than that in the (N-1)th layer image. In this case, after determining the range of comparative disparity values ​​for the corresponding pixels in the (N-1)th layer image, for pixels in the (N-1)th layer image that do not contain pixels from the Nth layer image, the determined range of comparative disparity values ​​can be used as a reference to determine one's own comparative disparity range. For example, ... Figure 6 As shown, the Nth layer image is a 2x2 image, and the (N-1)th layer image is a 4x4 image. When the disparity range of four pixels in the Nth layer image is determined (pixels a1, a2, a3, and a4 shown in the figure), the comparison disparity range of the corresponding four pixels in the (N-1)th layer image can be determined first (pixels b1, b2, b3, and b4 shown in the figure correspond to pixels a1, a2, a3, and a4). Then, when determining the comparison disparity range of other pixels in the (N-1)th layer image, adjacent pixels can be used... The comparison parallax range of each pixel is determined as its own comparison parallax range. Specifically, the comparison parallax range of pixel b1 in the diagram is used as the comparison parallax range of its neighboring pixels c1, c3, and c4; the comparison parallax range of pixel b2 is used as the comparison parallax range of its neighboring pixels c2, c5, and c6; the comparison parallax range of pixel b3 is used as the comparison parallax range of its neighboring pixels c7, c9, and c10; and the comparison parallax range of pixel b4 is used as the comparison parallax range of its neighboring pixels c8, c11, and c12. It should also be noted that... Figure 6 The Nth layer image is obtained by downsampling the (N-1)th layer image, and the sampling rule is uniform interval sampling. Under this sampling rule, pixels a1, a2, a3, a4 in the Nth layer image correspond to pixels b1, b2, b3, b4 in the (N-1)th layer image. In this case, Figure 6 This is merely one method to determine the comparison disparity range of other pixels (c1-c12) based on the comparison disparity value range of the already determined pixels (b1, b2, b3, b4) on the (N-1)th layer image. It is only one implementation method, and other rules can be used to determine it in actual applications, which are not limited here.

[0089] Optionally, specific implementations of the above embodiments are provided, such as... Figure 7 As shown, this method includes:

[0090] S3010, According to the sampling rules, determine the target pixel in the Nth layer image of the first pyramid that corresponds to each pixel in the (N-1)th layer image of the first pyramid.

[0091] The sampling rules for downsampling the (N-1)th layer image in the first pyramid differ, resulting in different target pixels determined in the Nth layer image compared to those in the (N-1)th layer image. Specifically, the sampling rules can be based on uniformly spaced sampling or partial region sampling of pixels in the original image, or on uniformly spaced sampling or partial region sampling of pixels in the processed original image. The following examples illustrate the methods for determining target pixels under these two different sampling rules.

[0092] Example 1: If the sampling rule is based on uniformly spaced sampling of pixels in the original image or sampling of pixels in a partial region, it means that pixels in the Nth layer of the first pyramid are correspondingly included in the (N-1)th layer of the first pyramid. In other words, there exist pixels in the (N-1)th layer of the first pyramid with the same pixel value as pixels in the Nth layer. Therefore, when determining the target pixel in the Nth layer of the first pyramid that corresponds to each pixel in the (N-1)th layer, the pixel with the same pixel value can be directly determined as the target pixel. For example, refer to... Figure 6 The diagram illustrates a pyramid. If the Nth layer image is a 2x2 image and the (N-1)th layer image is a 4x4 image, then in the (N-1)th layer image, we search for pixels whose pixel values ​​match those of each pixel in the Nth layer image. The pixel in the Nth layer image corresponding to the searched pixel is then taken as the target pixel in the Nth layer image of the first pyramid. For example, suppose... Figure 6 If the pixel values ​​of pixels b1, b2, b3, and b4 in the (N-1)th layer image are the same as the pixel values ​​of pixels a1, a2, a3, and a4 in the Nth layer image, then the target pixel in the Nth layer image corresponding to pixel b1 in the (N-1)th layer image is a1; the target pixel in the Nth layer image corresponding to pixel b2 in the (N-1)th layer image is a2; the target pixel in the Nth layer image corresponding to pixel b3 in the (N-1)th layer image is a3; and the target pixel in the Nth layer image corresponding to pixel b4 in the (N-1)th layer image is a4.

[0093] For pixels in the (N-1)th layer image with inconsistent pixel values, such as pixels c1, c2, c3, c4, c5, c6, c7, c8, c9, c10, c11, and c12, pixels with similar pixel values ​​can be identified as target pixels. For example, if the pixel values ​​of pixels c1, b1, c3, and c4 in the (N-1)th layer image are similar to the pixel value of pixel a1 in the Nth layer image, then pixel a1 will be considered the target pixel for pixels c1, b1, c3, and c4; and the pixel values ​​of pixels c2, b2, c5, and c6 in the (N-1)th layer image will also be similar. If the pixel value of pixel a2 in the Nth layer image is similar to that of pixel a2, then pixel a2 is selected as the target pixel of pixels c2, b2, c5, and c6. If the pixel values ​​of pixels c7, b3, c9, and c10 in the (N-1)th layer image are similar to those of pixel a3 in the Nth layer image, then pixel a3 is selected as the target pixel of pixels c7, b3, c9, and c10. If the pixel values ​​of pixels c8, b4, c11, and c12 in the (N-1)th layer image are similar to those of pixel a4 in the Nth layer image, then pixel a4 is selected as the target pixel of pixels c7, b3, c9, and c10.

[0094] Optionally, for pixels in the (N-1)th layer image with inconsistent pixel values, the target pixel can also be determined by dividing the image into regions. For example, using the pixel corresponding to the target pixel as a reference point, the (N-1)th layer image can be divided into regions. Figure 6 In the example, the target pixels are first determined to be a1, a2, a3, and a4. Then, the pixels corresponding to these target pixels in the (N-1)th layer image are b1, b2, b3, and b4. When dividing the image into regions, a region D1 can be defined using pixel b1 as the reference point. All other pixels in region D1 (c1, c3, c4) correspond to the target pixel a1. Similarly, a region D2 can be defined using pixel b2 as the reference point, and all other pixels in region D2 (c2, c5, c6) correspond to the target pixel a2. A region D3 can be defined using pixel b3 as the reference point, and all other pixels in region D3 (c7, c9, c10) correspond to the target pixel a3. Finally, a region D4 can be defined using pixel b4 as the reference point, and all other pixels in region D4 (c8, c11, c12) correspond to the target pixel a4. This example of region division is merely illustrative; other division rules can be used in practice, and this example is not limited to this one.

[0095] Example 2: If the sampling rule is based on uniform interval sampling or partial region sampling of pixels after processing the original image, this sampling rule means that pixels in the Nth layer of the first pyramid are unlikely to be included in the (N-1)th layer of the first pyramid. In other words, there are no pixels in the (N-1)th layer of the first pyramid with the same pixel value as pixels in the Nth layer. Under this type of sampling rule, the target pixel can be determined by comparing similar pixel values ​​or by dividing the region. These two methods can be found in the explanation of Example 1 above, and will not be elaborated here.

[0096] S3011, based on the disparity map of the Nth layer image in the first pyramid, the disparity value corresponding to the target pixel in the Nth layer image in the first pyramid is used as the comparison disparity value of the corresponding pixel in the (N-1)th layer image in the first pyramid.

[0097] When the electronic device obtains the disparity map of the Nth layer image in the first pyramid and the target pixels of the Nth layer image in the first pyramid based on the aforementioned steps, it can determine the disparity value of each target pixel in the Nth layer image in the first pyramid, as well as the pixel corresponding to each target pixel in the (N-1)th layer image in the first pyramid. Subsequently, the electronic device can use the disparity value of each target pixel in the Nth layer image in the first pyramid as the comparison disparity value of the corresponding pixel in the (N-1)th layer image. Optionally, the electronic device can also correct the disparity value of each target pixel when determining the disparity value of each target pixel in the Nth layer image in the first pyramid, such as through weighted correction, and then use the corrected disparity value of each target pixel in the Nth layer image in the first pyramid as the comparison disparity value of the corresponding pixel in the (N-1)th layer image. For example, as... Figure 6 As shown, in the first pyramid, the target pixels in the Nth layer of the image are a1, a2, a3, and a4. The pixels corresponding to target pixel a1 in the (N-1)th layer of the image are b1, c1, c3, and c4. Therefore, the disparity value or corrected disparity value of target pixel a1 is used as the comparison disparity value for pixels b1, c1, c3, and c4 respectively. Similarly, the pixels corresponding to target pixel a2 in the (N-1)th layer of the image are b2, c2, c5, and c6. Therefore, the disparity value or corrected disparity value of target pixel a2 is used as the comparison disparity value for pixels b2, c2, and c6 respectively. The comparison disparity values ​​of c5 and c6 are respectively; in the (N-1)th layer image, the pixels corresponding to the target pixel a3 are b3, c7, c9, and c10, then the disparity value or the corrected disparity value of the target pixel a3 is used as the comparison disparity value of each pixel b3, c7, c9, and c10; in the (N-1)th layer image, the pixels corresponding to the target pixel a4 are b4, c8, c11, and c12, then the disparity value or the corrected disparity value of the target pixel a4 is used as the comparison disparity value of each pixel b4, c8, c11, and c12.

[0098] S3012, based on the comparison disparity value of each pixel in the (N-1)th layer image of the first pyramid and the first preset value, determine the range of comparison disparity values ​​of each pixel in the (N-1)th layer image of the first pyramid.

[0099] The first preset value can be any positive or negative integer, that is, it can be a plus or minus value.

[0100] When the electronic device determines the comparison disparity value of each pixel in the (N-1)th layer of the first pyramid image based on the aforementioned steps, it can further select a first preset value and perform calculations, such as addition and subtraction, on the comparison disparity value of each pixel to obtain a numerical range containing the comparison disparity value, thereby obtaining the range of comparison disparity values ​​for each pixel in the (N-1)th layer of the first pyramid image. For example, taking a single pixel as an example... Figure 5 The diagram shows that d0 is the comparison disparity value of a pixel in the (N-1)th layer image. Assuming this comparison disparity value is 0, a first preset value of 5 is selected. Then, by adding and subtracting 5 from this comparison disparity value, the resulting comparison disparity value range D is {-5, 5}. It should be noted that the first preset value for different pixels can be the same or different; that is, the comparison disparity value range for different pixels can be the same or different.

[0101] S302, based on the multiple cost values ​​corresponding to each pixel in the (N-1)th layer of the first pyramid image, obtain multiple first target disparity values ​​corresponding to each pixel in the (N-1)th layer of the first pyramid image; take the first target disparity value among the multiple first target disparity values ​​corresponding to each pixel in the (N-1)th layer of the first pyramid image as the target disparity value of the corresponding pixel in the (N-1)th layer of the first pyramid image.

[0102] This embodiment relates to a method for determining the target disparity value of each pixel in the (N-1)th layer image. Specifically, when the electronic device determines the range of comparative disparity values ​​of each pixel in the (N-1)th layer image of the first pyramid based on the aforementioned step S301, it can further use any first target disparity value within the range of comparative disparity values ​​corresponding to each pixel as the target disparity value of the corresponding pixel. Optionally, the electronic device can also filter out the first target disparity value that meets the preset conditions within the range of comparative disparity values ​​corresponding to each pixel as the target disparity value of the corresponding pixel. Optionally, the electronic device can also perform a weighted summation or average operation on all the first target disparity values ​​within the range of comparative disparity values ​​corresponding to each pixel, and use the disparity value obtained by the weighted summation or average operation as the target disparity value of the corresponding pixel.

[0103] S303, determine the disparity map of the N-1 layer image in the first pyramid based on the target disparity value of the corresponding pixel in the N-1 layer image of the first pyramid.

[0104] When the electronic device acquires the target disparity value of a corresponding pixel in the (N-1)th layer image of the first pyramid, the target disparity value of the corresponding pixel can be one or more. If any pixel in the (N-1)th layer image of the first pyramid has one target disparity value, the electronic device directly uses the target disparity value corresponding to any pixel in the (N-1)th layer image of the first pyramid as the disparity value of the corresponding pixel in the (N-1)th layer image of the first pyramid. If any pixel in the (N-1)th layer image of the first pyramid has two or more target disparity values, the electronic device uses the smallest target disparity value among the target disparity values ​​corresponding to any pixel in the (N-1)th layer image of the first pyramid as the disparity value of the corresponding pixel in the (N-1)th layer image of the first pyramid. If any pixel in the (N-1)th layer of the first pyramid image has two or more target disparity values, the electronic device can further use the minimum cost value among all the target disparity values ​​of any pixel in the (N-1)th layer of the first pyramid image as the second target cost value, and use the target disparity value of any pixel in the (N-1)th layer of the first pyramid image corresponding to the second target cost value as the disparity value of the corresponding pixel in the (N-1)th layer of the first pyramid image. When the target disparity values ​​of each pixel in the (N-1)th layer of the first pyramid image are determined based on the above method, the disparity map of the (N-1)th layer of the first pyramid image can be obtained.

[0105] In one embodiment, a method for constructing a first pyramid based on a first image is provided, namely, an implementation of the above-mentioned S101 "constructing a first pyramid by downsampling the first image N times according to sampling rules", such as... Figure 8 As shown, this implementation method includes:

[0106] S401, according to the sampling rules, the first image is downsampled N times in sequence to obtain N layers of third images with different resolutions.

[0107] When the electronic device acquires the first image, it can select a sampling rule to perform N downsampling operations on the first image sequentially. Each downsampling operation uses the same sampling frequency, resulting in N layers of third images with different resolutions. For example, assuming the first image has a resolution of 1024*512, when N is 4, the first image is downsampled 4 times sequentially, resulting in 4 layers of third images with different resolutions: a third image with a resolution of 128*64, a third image with a resolution of 64*32, a third image with a resolution of 32*16, and a third image with a resolution of 16*8. It should be noted that the sampling rule selected in this embodiment can be determined according to actual sampling requirements. For example, the sampling rule can be based on uniformly spaced sampling of pixels in the original image or sampling of pixels in a partial region, or it can be based on uniformly spaced sampling of pixels in the processed original image or sampling of pixels in a partial region.

[0108] S402, combine the first image and N layers of third images with different resolutions in descending order of resolution to construct the first pyramid.

[0109] The electronic device combines a first image and N layers of third images with different resolutions in descending order of resolution. Specifically, the lowest-resolution third images are used as the upper layers of the first pyramid, and the highest-resolution third images are used as the lower layers. The first pyramid is constructed in this way, with the highest layer being the lowest-resolution third image, the lowest-resolution image being the first image, and the other layers arranged from highest to lowest resolution. For example, assuming the first image has a resolution of 1024*512, when N is 4, the first image is downsampled 4 times to obtain 4 layers of third images with different resolutions. These 4 layers of third images with different resolutions, along with the first image, form a 5-layer pyramid. Figure 9 As shown, the resolution of the third image of the fourth layer in the first pyramid is 16*8, the resolution of the third image of the third layer in the first pyramid is 32*16, the resolution of the third image of the second layer in the first pyramid is 64*32, the resolution of the third image of the first layer in the first pyramid is 128*64, and the resolution of the image of the 0th layer (first image) in the first pyramid is 1024*512.

[0110] Correspondingly, a method for constructing a second pyramid based on a second image is also provided, namely, an implementation of the above-mentioned S101 "constructing a second pyramid by downsampling the second image N times according to the sampling rules", such as... Figure 10 As shown, the steps include:

[0111] S501, according to the sampling rules, the second image is downsampled N times in sequence to obtain N layers of fourth images with different resolutions.

[0112] When the electronic device acquires the second image, it can select a sampling rule to perform N downsampling operations on the second image sequentially, with the same sampling frequency for each downsampling operation, resulting in N layers of fourth images with different resolutions. For example, assuming the resolution of the second image is 1024*512, when N is 4, the second image is downsampled 4 times sequentially, resulting in 4 layers of fourth images with different resolutions: a fourth image with a resolution of 128*64, a fourth image with a resolution of 64*32, a fourth image with a resolution of 32*16, and a fourth image with a resolution of 16*8. It should be noted that the sampling rule selected in this embodiment can be determined according to actual sampling requirements. For example, the sampling rule can be based on uniform interval sampling or partial region sampling of pixels in the original image, or it can be based on uniform interval sampling or partial region sampling of pixels in the processed original image.

[0113] S502, combine the second image and the N layers of fourth images with different resolutions in descending order of resolution to construct the second pyramid.

[0114] The electronic device combines a second image and N layers of fourth images with different resolutions, arranged in descending order of resolution. Specifically, the lower-resolution fourth images are used as the upper layers of the second pyramid, and the higher-resolution fourth images are used as the lower layers. The second pyramid is constructed in this way, with the highest layer being the lowest-resolution fourth image, the lowest-resolution image being the second image, and the other layers arranged in ascending order of resolution. For example, assuming the second image has a resolution of 1024*512, when N is 4, the second image is downsampled 4 times to obtain 4 layers of fourth images with different resolutions. These 4 layers of fourth images and the second image are then used to construct a 5-layer pyramid. Figure 9 As shown, the resolution of the fourth image of the fourth layer in the second pyramid is 16*8, the resolution of the fourth image of the third layer in the second pyramid is 32*16, the resolution of the fourth image of the second layer in the second pyramid is 64*32, the resolution of the fourth image of the first layer in the second pyramid is 128*64, and the resolution of the image (second image) of the 0th layer in the second pyramid is 1024*512.

[0115] Optionally, one implementation of S102 is provided, namely, S102 "performs stereo matching calculation on the Nth layer image in the first pyramid and the Nth layer image in the second pyramid to obtain the disparity map of the Nth layer image in the first pyramid", such as Figure 11 As shown, it includes:

[0116] S601, calculate the cost of the Nth layer image in the first pyramid and the Nth layer image in the second pyramid to obtain multiple cost values ​​corresponding to each pixel in the Nth layer image in the first pyramid.

[0117] In this embodiment, a corresponding cost calculation method can be used to calculate the cost of the Nth layer image in the first pyramid and the Nth layer image in the second pyramid, so as to obtain multiple cost values ​​corresponding to each pixel in the Nth layer image in the first pyramid. Similarly, multiple cost values ​​corresponding to each pixel in the Nth layer image in the second pyramid can also be obtained.

[0118] S602, take the minimum value among the multiple values ​​corresponding to each pixel in the Nth layer image of the first pyramid as the target value corresponding to the corresponding pixel; obtain the disparity value corresponding to each pixel in the Nth layer image of the first pyramid according to the target value corresponding to each pixel in the Nth layer image of the first pyramid, so as to obtain the disparity map of the Nth layer image of the first pyramid.

[0119] When the electronic device obtains multiple values ​​corresponding to each pixel in the Nth layer image of the first pyramid, it can further filter the multiple values ​​corresponding to each pixel. Specifically, the minimum value is selected, and the corresponding disparity value is obtained directly based on the minimum value as the disparity value of the corresponding pixel, thus obtaining the disparity map of the Nth layer image in the first pyramid.

[0120] Optionally, the electronic device can also perform cost calculation, cost aggregation, disparity estimation, and disparity optimization sequentially on the Nth layer image in the first pyramid and the Nth layer image in the second pyramid to obtain the disparity map of the Nth layer image in the first pyramid, i.e., the disparity map of the Nth layer image in the first pyramid; or it can obtain the disparity map of the Nth layer image in the second pyramid accordingly.

[0121] Optionally, the electronic device can use the SSBM or AD-Census algorithm to perform stereo matching calculations on the Nth layer image in the first pyramid and the Nth layer image in the second pyramid to obtain the disparity map of the Nth layer image in the first pyramid.

[0122] Based on all the above embodiments, the method for determining image parallax described in this embodiment is illustrated by way of example:

[0123] Step 1: Assuming the first image has a height of H and a width of W, and the second image has a height of H and a width of W, downsample the first image N times to construct the first pyramid, and downsample the second image N times to construct the second pyramid. Then, the height of the Nth layer of the first pyramid is H / (2^N), the width is W / (2^N), and the 0th layer image is the first image; the height of the Nth layer of the second pyramid is H / (2^N), the width is W / (2^N), and the 0th layer image is the second image.

[0124] Step 2: Perform a complete stereo matching process (e.g., cost calculation, cost aggregation, disparity estimation, and disparity optimization) based on the Nth layer image in the first pyramid and the Nth layer image in the second pyramid. Specifically, stereo matching can be implemented using algorithms such as sgbm and AD-census to obtain the disparity map of the Nth layer image in the first pyramid and the disparity map of the Nth layer image in the second pyramid.

[0125] Step 3: Perform cost calculations on the (N-1)th layer image in the first pyramid and the (N-1)th layer image in the second pyramid to obtain multiple cost values ​​corresponding to each pixel in the (N-1)th layer image in the first pyramid and multiple cost values ​​corresponding to each pixel in the (N-1)th layer image in the second pyramid.

[0126] Step 4: Based on the disparity values ​​corresponding to each pixel in the disparity map of the Nth layer image in the first pyramid and the first preset value, determine the range of comparative disparity values ​​for each pixel in the (N-1)th layer image of the first pyramid. Obtain multiple first target disparity values ​​corresponding to each pixel in the (N-1)th layer image of the first pyramid based on multiple cost values. Then, take the first target disparity value from the range of comparative disparity values ​​for the corresponding pixel in the (N-1)th layer image of the first pyramid as the target disparity value for the corresponding pixel in the (N-1)th layer image of the first pyramid. Finally, determine the disparity map of the (N-1)th layer image of the first pyramid based on the target disparity value of the corresponding pixel in the (N-1)th layer image of the first pyramid. Determine the disparity map of the (N-1)th layer image of the second pyramid using the above method.

[0127] Step 5: Repeat the methods described in steps 2-4 until reaching layer 0. Specifically, based on the disparity values ​​of each pixel in the disparity map of layer 1 of the first pyramid and a first preset value, determine the range of comparison disparity values ​​for each pixel in layer 0 of the first pyramid. Then, take the first target disparity value from the range of comparison disparity values ​​for the corresponding pixel in layer 0 of the first pyramid as the target disparity value for that pixel. Finally, determine the disparity map of layer 0 of the first pyramid based on the target disparity value of the corresponding pixel in layer 0 of the first pyramid. The disparity map of layer 0 of the second pyramid is then determined using the same method.

[0128] In the above method, during the processing of images from layer N to layer 0 in any pyramid, the sum of the cost aggregation times is approximately equal to the time spent calculating costs twice on the layer 0 image. However, if complete stereo matching is performed on layer 0, cost calculation only occupies a small portion of the time; the majority of the time is spent on cost aggregation and cost calculation. In other words, the above method only performs cost calculations on images other than layer N, significantly reducing the time required for binocular matching. Furthermore, the search near prior disparity is a cyclical process, which can be easily accelerated using the CPU, thus improving the overall frame rate of binocular stereo matching.

[0129] It should be understood that although the steps in the flowchart are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order constraint on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the diagram may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0130] In one embodiment, such as Figure 12 As shown, an image parallax determination device is provided, comprising:

[0131] Construction module 11 is used to construct a first pyramid by performing N downsampling operations on a first image according to the sampling rules, and to construct a second pyramid by performing N downsampling operations on a second image according to the sampling rules; wherein, N is an integer greater than or equal to 1, the top layer of the first pyramid is the Nth layer, the bottom layer of the first pyramid is the 0th layer, the resolution of the images from the 0th layer to the Nth layer in the first pyramid gradually decreases, and the image from the 0th layer in the first pyramid is the first image; the top layer of the second pyramid is the Nth layer, the bottom layer of the second pyramid is the 0th layer, the resolution of the images from the 0th layer to the Nth layer in the second pyramid gradually decreases, and the image from the 0th layer in the second pyramid is the second image;

[0132] Matching module 12 is used to perform stereo matching calculations on the Nth layer image in the first pyramid and the Nth layer image in the second pyramid to obtain a disparity map of the Nth layer image in the first pyramid.

[0133] The calculation module 13 is used to perform cost calculation on the image at layer 0 of the first pyramid and the image at layer 0 of the second pyramid to obtain multiple cost values ​​corresponding to each pixel in the image at layer 0 of the first pyramid.

[0134] The determining module 14 is used to obtain the disparity map of the image of the Nth layer in the first pyramid based on the disparity map of the image of the Nth layer in the first pyramid and the multiple cost values ​​corresponding to each pixel in the image of the 0th layer in the first pyramid.

[0135] For specific limitations regarding the aforementioned image disparity determination device, please refer to the limitations of the image disparity determination method described above, which will not be repeated here. Each module in the image disparity determination device can be implemented entirely or partially through software, hardware, or a combination thereof. The aforementioned devices, modules, or units can be embedded in or independent of the processor in the electronic device in hardware form, or stored in the memory of the electronic device in software form, so that the processor can call and execute the operations corresponding to the aforementioned devices or modules.

[0136] In one embodiment, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0137] A first pyramid is constructed by downsampling the first image N times according to the sampling rules, and a second pyramid is constructed by downsampling the second image N times according to the sampling rules; wherein, N is an integer greater than or equal to 1, the top layer of the first pyramid is the Nth layer, the bottom layer of the first pyramid is the 0th layer, the resolution of the images from the 0th layer to the Nth layer in the first pyramid gradually decreases, and the image from the 0th layer in the first pyramid is the first image; the top layer of the second pyramid is the Nth layer, the bottom layer of the second pyramid is the 0th layer, the resolution of the images from the 0th layer to the Nth layer in the second pyramid gradually decreases, and the image from the 0th layer in the second pyramid is the second image;

[0138] Stereo matching calculations are performed on the Nth layer image of the first pyramid and the Nth layer image of the second pyramid to obtain the disparity map of the Nth layer image of the first pyramid;

[0139] Cost calculations are performed on the images at layer 0 of the first pyramid and the images at layer 0 of the second pyramid to obtain multiple cost values ​​corresponding to each pixel in the image at layer 0 of the first pyramid.

[0140] Based on the disparity map of the Nth layer image in the first pyramid and the multiple cost values ​​corresponding to each pixel in the 0th layer image in the first pyramid, the disparity map of the 0th layer image in the first pyramid is obtained.

[0141] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0142] A first pyramid is constructed by downsampling the first image N times according to the sampling rules, and a second pyramid is constructed by downsampling the second image N times according to the sampling rules; wherein, N is an integer greater than or equal to 1, the top layer of the first pyramid is the Nth layer, the bottom layer of the first pyramid is the 0th layer, the resolution of the images from the 0th layer to the Nth layer in the first pyramid gradually decreases, and the image from the 0th layer in the first pyramid is the first image; the top layer of the second pyramid is the Nth layer, the bottom layer of the second pyramid is the 0th layer, the resolution of the images from the 0th layer to the Nth layer in the second pyramid gradually decreases, and the image from the 0th layer in the second pyramid is the second image;

[0143] Stereo matching calculations are performed on the Nth layer image of the first pyramid and the Nth layer image of the second pyramid to obtain the disparity map of the Nth layer image of the first pyramid;

[0144] Cost calculations are performed on the images at layer 0 of the first pyramid and the images at layer 0 of the second pyramid to obtain multiple cost values ​​corresponding to each pixel in the image at layer 0 of the first pyramid.

[0145] Based on the disparity map of the Nth layer image in the first pyramid and the multiple cost values ​​corresponding to each pixel in the 0th layer image in the first pyramid, the disparity map of the 0th layer image in the first pyramid is obtained.

[0146] In one embodiment, a computer program product is provided, which, when executed by a processor, performs the following steps:

[0147] A first pyramid is constructed by downsampling the first image N times according to the sampling rules, and a second pyramid is constructed by downsampling the second image N times according to the sampling rules; wherein, N is an integer greater than or equal to 1, the top layer of the first pyramid is the Nth layer, the bottom layer of the first pyramid is the 0th layer, the resolution of the images from the 0th layer to the Nth layer in the first pyramid gradually decreases, and the image from the 0th layer in the first pyramid is the first image; the top layer of the second pyramid is the Nth layer, the bottom layer of the second pyramid is the 0th layer, the resolution of the images from the 0th layer to the Nth layer in the second pyramid gradually decreases, and the image from the 0th layer in the second pyramid is the second image;

[0148] Stereo matching calculations are performed on the Nth layer image of the first pyramid and the Nth layer image of the second pyramid to obtain the disparity map of the Nth layer image of the first pyramid;

[0149] Cost calculations are performed on the images at layer 0 of the first pyramid and the images at layer 0 of the second pyramid to obtain multiple cost values ​​corresponding to each pixel in the image at layer 0 of the first pyramid.

[0150] Based on the disparity map of the Nth layer image in the first pyramid and the multiple cost values ​​corresponding to each pixel in the 0th layer image in the first pyramid, a disparity map of the 0th layer image in the first pyramid is obtained. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0151] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0152] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for determining image parallax, wherein the method for determining image parallax is applied to an electronic device, characterized in that, include: A first pyramid is constructed by downsampling the first image N times according to the sampling rules, and a second pyramid is constructed by downsampling the second image N times according to the sampling rules; wherein, N is an integer greater than or equal to 1, the top layer of the first pyramid is the Nth layer, the bottom layer of the first pyramid is the 0th layer, the resolution of the images from the 0th layer to the Nth layer in the first pyramid gradually decreases, and the image from the 0th layer in the first pyramid is the first image; the top layer of the second pyramid is the Nth layer, the bottom layer of the second pyramid is the 0th layer, the resolution of the images from the 0th layer to the Nth layer in the second pyramid gradually decreases, and the image from the 0th layer in the second pyramid is the second image; Stereo matching calculations are performed on the Nth layer image in the first pyramid and the Nth layer image in the second pyramid to obtain a disparity map of the Nth layer image in the first pyramid; the stereo matching calculation includes cost calculation, cost aggregation, disparity estimation and disparity optimization; Cost calculations are performed on the images at layer 0 of the first pyramid and the images at layer 0 of the second pyramid to obtain multiple cost values ​​corresponding to each pixel in the image at layer 0 of the first pyramid; only cost calculations are performed on the images at other layers of the first pyramid except for the image at layer N. Based on the disparity map of the Nth layer image in the first pyramid and the multiple cost values ​​corresponding to each pixel in the 0th layer image in the first pyramid, the disparity map of the 0th layer image in the first pyramid is obtained. The step of performing stereo matching calculations on the Nth layer image of the first pyramid and the Nth layer image of the second pyramid to obtain the disparity map of the Nth layer image of the first pyramid includes: Cost calculations are performed on the Nth layer image of the first pyramid and the Nth layer image of the second pyramid to obtain multiple cost values ​​corresponding to each pixel in the Nth layer image of the first pyramid; The minimum value among the multiple values ​​corresponding to each pixel in the Nth layer of the first pyramid is taken as the target value corresponding to the pixel. Based on the target cost value corresponding to each pixel in the Nth layer image of the first pyramid, the disparity value corresponding to each pixel in the Nth layer image of the first pyramid is obtained, so as to obtain the disparity map of the Nth layer image of the first pyramid.

2. The method for determining image parallax according to claim 1, characterized in that, Based on the disparity map of the Nth layer image of the first pyramid and the multiple cost values ​​corresponding to each pixel in the 0th layer image of the first pyramid, the disparity map of the 0th layer image of the first pyramid is obtained, including: Cost calculations are performed on the (N-1)th layer image of the first pyramid and the (N-1)th layer image of the second pyramid to obtain multiple cost values ​​corresponding to each pixel in the (N-1)th layer image of the first pyramid; Based on the disparity map of the Nth layer image in the first pyramid and the multiple cost values ​​corresponding to each pixel in the (N-1)th layer image in the first pyramid, the disparity map of the (N-1)th layer image in the first pyramid is determined. Based on the disparity map of the (N-1)th layer image in the first pyramid and the multiple cost values ​​corresponding to each pixel in the 0th layer image in the first pyramid, the disparity map of the 0th layer image in the first pyramid is obtained.

3. The method for determining image parallax according to claim 1 or 2, characterized in that, The step of downsampling the first image N times according to the sampling rules to construct the first pyramid includes: The first image is downsampled N times according to the sampling rules to obtain N layers of third images with different resolutions; The first image and the N layers of third images with different resolutions are combined in descending order of resolution to construct the first pyramid; The step of downsampling the second image N times according to the sampling rule to construct the second pyramid includes: The second image is downsampled N times according to the sampling rule to obtain N layers of fourth images with different resolutions; The second image and the N layers of fourth images with different resolutions are combined in descending order of resolution to construct the second pyramid.

4. The method for determining image parallax according to claim 2, characterized in that, The step of determining the disparity map of the (N-1)th layer image in the first pyramid based on the disparity map of the Nth layer image in the first pyramid and the multiple cost values ​​corresponding to each pixel in the (N-1)th layer image in the first pyramid includes: Based on the disparity map of the Nth layer image in the first pyramid and the first preset value, determine the range of comparative disparity values ​​for each pixel in the (N-1)th layer image of the first pyramid; Multiple first target disparity values ​​are obtained for each pixel in the (N-1)th layer of the first pyramid image based on multiple cost values. The first target disparity value among the multiple first target disparity values ​​corresponding to each pixel in the N-1th layer image of the first pyramid, which falls within the range of the comparison disparity values ​​of the corresponding pixel in the N-1th layer image of the first pyramid, is taken as the target disparity value of the corresponding pixel in the N-1th layer image of the first pyramid. Based on the target disparity value of the corresponding pixel in the (N-1)th layer image of the first pyramid, determine the disparity map of the (N-1)th layer image of the first pyramid.

5. The method for determining image parallax according to claim 4, characterized in that, The step of determining the disparity map of the (N-1)th layer image in the first pyramid based on the target disparity value of the corresponding pixel in the (N-1)th layer image in the first pyramid includes: When any pixel in the (N-1)th layer of the first pyramid has two or more target disparity values, the minimum target disparity value among the target disparity values ​​corresponding to any pixel in the (N-1)th layer of the first pyramid shall be taken as the disparity value of the corresponding pixel in the (N-1)th layer of the first pyramid. When any pixel in the (N-1)th layer of the first pyramid has a target disparity value, the target disparity value corresponding to any pixel in the (N-1)th layer of the first pyramid is taken as the disparity value of the corresponding pixel in the (N-1)th layer of the first pyramid.

6. The method for determining image parallax according to claim 4, characterized in that, The step of determining the range of comparative disparity values ​​for each pixel in the (N-1)th layer of the first pyramid image based on the disparity map of the Nth layer image in the first pyramid includes: According to the sampling rule, target pixels corresponding to each pixel in the (N-1)th layer of the first pyramid are determined in the Nth layer image of the first pyramid; Based on the disparity map of the Nth layer image in the first pyramid, the disparity value corresponding to the target pixel in the Nth layer image in the first pyramid is used as the comparison disparity value of the corresponding pixel in the (N-1)th layer image in the first pyramid. Based on the comparison disparity value of each pixel in the (N-1)th layer image of the first pyramid and the first preset value, the range of comparison disparity values ​​of each pixel in the (N-1)th layer image of the first pyramid is determined.

7. The method for determining image parallax according to claim 1, characterized in that, The step of performing stereo matching calculations on the Nth layer image of the first pyramid and the Nth layer image of the second pyramid to obtain the disparity map of the Nth layer image of the first pyramid includes: The SSBM or AD-Census algorithm is used to perform stereo matching calculations on the Nth layer image in the first pyramid and the Nth layer image in the second pyramid to obtain the disparity map of the Nth layer image in the first pyramid.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images