Methods, processing circuitry, storage media, and program products for stereo matching

Abstract

A method, processing circuit, storage medium and program product for stereo matching. The method is performed by a processing circuit comprising an on-chip memory, the method comprising: determining, by the processing circuit, a first dynamic matching cost corresponding to a predetermined disparity range between a first image block of a plurality of image blocks of a first image (250) and a second image (260), wherein the first image (250) and the second image (260) form a pair of binocular vision images. Further, the processing circuit stores, in the on-chip memory, edge cost information corresponding to a set of edge pixels in the first dynamic matching cost, wherein the set of edge pixels comprises pixels in the first image block adjacent to a second image block of the plurality of image blocks. Accordingly, the processing circuit determines a second dynamic matching cost corresponding to the predetermined disparity range between the second image block and the second image (260) based on the edge cost information obtained from the on-chip memory. The processing circuit determines a disparity map of the first image (250) and the second image (260) based on at least the first dynamic matching cost and the second dynamic matching cost. In this way, the method can reduce the usage of the on-chip memory, and thus can reduce the cost of the processing circuit while maintaining the computational efficiency.

Description

Technical Field

[0001] The embodiments of this disclosure primarily relate to the field of computers, and more specifically, to methods, processing circuits, storage media, and program products for stereo matching. Background Technology

[0002] With the development of computer technology, the remarkable performance of binocular stereo vision in scene depth perception, image recognition, image preprocessing, autonomous driving, VR / AR, and other applications has made it a hot topic in various academic research fields. Stereo matching, also known as disparity estimation, determines the disparity map between two images by analyzing a pair of left and right images captured at the same time. Improving the computational efficiency of stereo matching has become a current focus of attention. Summary of the Invention

[0003] The embodiments of this disclosure provide a stereo matching scheme that can be used to facilitate the determination of disparity maps between binocular visual images, and can be widely applied in many fields such as image recognition, autonomous driving, virtual reality, and augmented reality.

[0004] In a first aspect of this disclosure, a method for stereo matching is provided, which can be executed by processing circuitry including on-chip memory. The method includes: determining, by the processing circuitry, a first dynamic matching cost corresponding to a predetermined disparity range between a first image block of a plurality of image blocks of a first image and a second image, the first image and the second image forming a pair of binocular visual images; storing in the on-chip memory edge cost information corresponding to a set of edge pixels in the first dynamic matching cost, wherein the set of edge pixels includes pixels in the first image block adjacent to the second image block of the plurality of image blocks; determining a second dynamic matching cost corresponding to the predetermined disparity range between the second image block and the second image based on the edge cost information obtained from the on-chip memory; and providing the first dynamic matching cost and the second dynamic matching cost for determining a disparity map of the first image and the second image.

[0005] In this way, embodiments of the present disclosure can reduce the size of on-chip memory required in calculating the matching cost by dividing the image into multiple image blocks and storing the matching cost of edge pixels in on-chip memory, thereby reducing the cost of processing circuitry while maintaining computational efficiency.

[0006] In some embodiments of the first aspect, determining the first dynamic matching cost includes: determining a static matching cost between the first image patch and the second image corresponding to a predetermined disparity range; and determining a dynamic matching cost of the first pixel based on the static matching cost of the first pixel in the first image patch and the dynamic matching cost of at least one associated pixel associated with the first pixel, wherein the at least one associated pixel is determined based on at least one dynamic programming direction.

[0007] In this way, embodiments of the present disclosure can determine the dynamic matching cost between pixels based on the static matching cost, thereby improving the accuracy of matching.

[0008] In some embodiments of the first aspect, the edge cost information includes the dynamic matching cost of the edge pixel associated with at least one dynamic programming direction.

[0009] In this way, embodiments of the present disclosure can store the dynamic matching cost of edge pixels in at least one planning direction in on-chip memory, thereby improving the basic data for the calculation of dynamic cost in the planning direction in subsequent image blocks, thereby reducing the storage requirements of on-chip memory.

[0010] In some embodiments of the first aspect, determining the static matching cost includes: performing edge enhancement on a first image and a second image respectively to obtain a first enhanced image and a second enhanced image; determining a first pixel cost associated with a plurality of pixels in the first image based on the first enhanced image and the second enhanced image; determining a second pixel cost of the plurality of pixels based on the first image and the second image; determining a weighted pixel cost of the plurality of pixels based on a weighted sum of the first pixel cost and the second pixel cost; and determining a static matching cost between the first image and the second image based on the weighted pixel cost.

[0011] In this way, embodiments of the present disclosure can more accurately determine the static matching cost between pixels by utilizing a combination of the static matching costs of the edge-enhanced image and the original image.

[0012] In some embodiments of the first aspect, determining the first pixel cost includes: performing subpixel interpolation on the first enhanced image and the second enhanced image respectively to obtain a first interpolated image and a second interpolated image; and determining the first pixel cost associated with a plurality of pixels in the first image based on the first interpolated image and the second interpolated image.

[0013] In this way, embodiments of the present disclosure can more accurately assess the degree of difference between pixels through subpixel interpolation.

[0014] In some embodiments of the first aspect, determining a first pixel cost associated with a plurality of pixels in the first image based on a first interpolated image and a second interpolated image includes: for a second pixel among the plurality of pixels, determining a first maximum gray value and a first minimum gray value based on the gray value of the second pixel and the gray values ​​of interpolated pixels adjacent to the second pixel; for a third pixel in the second image that has a specific disparity with the second pixel, determining a second maximum gray value and a second minimum gray value based on the gray value of the third pixel and the gray values ​​of interpolated pixels adjacent to the third pixel; and determining a first pixel cost corresponding to a specific disparity of the second pixel based on a first difference between the gray value of the second pixel and the second maximum gray value, a second difference between the second minimum gray value and the gray value of the second pixel, a third difference between the gray value of the third pixel and the first maximum gray value, and a fourth difference between the first minimum gray value and the gray value of the third pixel.

[0015] In this way, embodiments of the present disclosure can take into account the continuity of pixels, thereby enabling a more accurate assessment of the degree of difference between pixels.

[0016] In some embodiments of the first aspect, determining the static matching cost between the first image and the second image based on the weighted pixel cost includes: determining an aggregation window associated with the fourth pixel in the first image; and determining the static matching cost of the fourth pixel based on the weighted pixel cost of each pixel within the aggregation window.

[0017] In this way, the embodiments of this disclosure can take into account the influence of differences in surrounding pixels, thereby improving the accuracy of pixel matching.

[0018] In some embodiments of the first aspect, at least one dynamic programming direction includes at least one of the following directions: a first direction indicated by direction vector (1, 0), a second direction indicated by direction vector (1, 1), or a third direction indicated by direction vector (0, 1).

[0019] In this way, the embodiments of this disclosure can take into account multiple dynamic programming directions and facilitate block processing and line-by-line scanning processing of pixels.

[0020] In some embodiments of the first aspect, the method further includes: filtering invalid pixels from a row of pixels based on a dynamic matching cost of a row of pixels in a second image block to obtain a set of candidate pixels, wherein the dynamic matching cost of the invalid pixels includes multiple matching costs corresponding to a predetermined disparity range, and the multiple matching costs include at least one interfering matching cost, wherein the difference between the at least one interfering matching cost and the minimum matching cost among the multiple matching costs is less than a predetermined threshold.

[0021] In this way, the embodiments of this disclosure can improve the accuracy of stereo matching through uniqueness detection.

[0022] In some embodiments of the first aspect, the method further includes: determining a set of candidate disparities corresponding to a set of candidate pixels based on a set of optimal matching costs, wherein the optimal matching cost is the minimum matching cost among a plurality of matching costs corresponding to a predetermined disparity range for the candidate pixels; and performing left-right consistency detection associated with the set of candidate pixels based on the set of candidate disparities to filter out falsely detected pixels in the set of candidate pixels.

[0023] In this way, the embodiments of this disclosure can improve the accuracy of stereo matching through left-right consistency detection.

[0024] In some embodiments of the first aspect, performing left-right consistency detection associated with a set of candidate pixels includes: determining a set of matching pixels in a second image corresponding to the set of candidate pixels based on a set of candidate disparities; and in response to determining that a target matching pixel in the set of matching pixels corresponds to a plurality of candidate pixels in the set of candidate pixels, determining a false-detection pixel from the plurality of candidate pixels based on a plurality of candidate disparities corresponding to the plurality of candidate pixels, wherein the difference between the candidate disparity corresponding to the false-detection pixel and the target candidate disparity is greater than a predetermined threshold, and the target candidate disparity corresponds to the candidate pixel with the smallest optimal matching cost value between the plurality of candidate pixels and the target matching pixel.

[0025] In this way, embodiments of the present disclosure can perform left-right consistency detection using only one image from the binocular vision images as a reference image, thereby further reducing computational and storage costs.

[0026] In some embodiments of the first aspect, a row of pixels is a first row of pixels, a set of matching pixels is a first set of matching pixels, and the on-chip memory includes a first row buffer and a second row buffer, the first row buffer and the second row buffer being configured to operate in a ping-pong buffer mode, wherein the first row buffer is used to store the disparity corresponding to the first set of matching pixels, and the second row buffer is used to store the disparity corresponding to the second set of matching pixels, the second set of matching pixels being determined based on the second row of pixels in the second image block.

[0027] In this way, embodiments of the present disclosure can utilize a ping-pong buffer mechanism to improve the utilization efficiency of on-chip memory, thereby supporting the parallelization of dynamic cost calculation and left-right consistency detection processes, and effectively improving the efficiency of stereo matching.

[0028] In some embodiments of the first aspect, the first image block and the second image block have overlapping regions, the first image block includes a first non-overlapping region, the second image block includes a second non-overlapping region, and a set of edge pixels includes: a set of pixels in the first non-overlapping region that are adjacent to the overlapping region, or a set of pixels in the overlapping region that are adjacent to the second non-overlapping region.

[0029] In this way, embodiments of the present disclosure improve the accuracy of the post-processing process by partially overlapping the segmented image blocks.

[0030] In some embodiments of the first aspect, a set of edge pixels is a first set of edge pixels, edge cost information is a first edge cost information, and providing a first dynamic matching cost and a second dynamic matching cost includes: storing in an on-chip memory a second edge cost information corresponding to a second set of edge pixels in the second dynamic matching cost, wherein the second set of edge pixels includes pixels in a second image block that are adjacent to a third image block among a plurality of image blocks; determining a third dynamic matching cost between the third image block and the second image corresponding to a predetermined disparity range based on the second edge cost information obtained from the on-chip memory; and providing the first dynamic matching cost, the second dynamic matching cost, and the third dynamic matching cost for determining a disparity map between the first image and the second image.

[0031] In this manner, embodiments of the present disclosure cache edge cost information for edge pixels of each intermediate image block to support dynamic matching cost calculation for the next image block. Consequently, embodiments of the present disclosure can reduce the width requirements of on-chip memory, thereby reducing the cost of the processing circuitry.

[0032] In a second aspect of this disclosure, a processing circuit is provided, including an on-chip memory. The processing circuit is configured to: determine a first dynamic matching cost corresponding to a predetermined disparity range between a first image block and a second image, which form a pair of binocular visual images; store in the on-chip memory edge cost information corresponding to a set of edge pixels in the first dynamic matching cost, wherein the set of edge pixels includes pixels in the first image block that are adjacent to the second image block among the plurality of image blocks; determine a second dynamic matching cost corresponding to the predetermined disparity range between the second image block and the second image based on the edge cost information obtained from the on-chip memory; and provide the first dynamic matching cost and the second dynamic matching cost for determining a disparity map between the first image and the second image.

[0033] In this way, embodiments of the present disclosure can reduce the size of on-chip memory required in calculating the matching cost by dividing the image into multiple image blocks and storing the matching cost of edge pixels in on-chip memory, thereby reducing the cost of processing circuitry while maintaining computational efficiency.

[0034] In some embodiments of the second aspect, determining the first dynamic matching cost includes: determining a static matching cost between the first image block and the second image corresponding to a predetermined disparity range; and determining a dynamic matching cost of the first pixel based on the static matching cost of the first pixel in the first image block and the dynamic matching cost of at least one associated pixel associated with the first pixel, wherein the at least one associated pixel is determined based on at least one dynamic programming direction.

[0035] In this way, embodiments of the present disclosure can determine the dynamic matching cost between pixels based on the static matching cost, thereby improving the accuracy of matching.

[0036] In some embodiments of the second aspect, the edge cost information includes the dynamic matching cost of the edge pixel associated with at least one dynamic programming direction.

[0037] In this way, embodiments of the present disclosure can store the dynamic matching cost of edge pixels in at least one planning direction in on-chip memory, thereby improving the basic data for the calculation of dynamic cost in the planning direction in subsequent image blocks, thereby reducing the storage requirements of on-chip memory.

[0038] In some embodiments of the second aspect, determining the static matching cost includes: performing edge enhancement on the first image and the second image respectively to obtain a first enhanced image and a second enhanced image; determining a first pixel cost associated with a plurality of pixels in the first image based on the first enhanced image and the second enhanced image; determining a second pixel cost of the plurality of pixels based on the first image and the second image; determining a weighted pixel cost of the plurality of pixels based on a weighted sum of the first pixel cost and the second pixel cost; and determining a static matching cost between the first image and the second image based on the weighted pixel cost.

[0039] In this way, embodiments of the present disclosure can more accurately determine the static matching cost between pixels by utilizing a combination of the static matching costs of the edge-enhanced image and the original image.

[0040] In some embodiments of the second aspect, determining the first pixel cost includes: performing subpixel interpolation on the first enhanced image and the second enhanced image respectively to obtain a first interpolated image and a second interpolated image; and determining a first pixel cost associated with a plurality of pixels in the first image based on the first interpolated image and the second interpolated image.

[0041] In this way, embodiments of the present disclosure can more accurately assess the degree of difference between pixels through subpixel interpolation.

[0042] In some embodiments of the second aspect, determining a first pixel cost associated with a plurality of pixels in the first image based on a first interpolated image and a second interpolated image includes: for a second pixel among the plurality of pixels, determining a first maximum gray value and a first minimum gray value based on the gray value of the second pixel and the gray values ​​of interpolated pixels adjacent to the second pixel; for a third pixel in the second image that has a specific disparity with the second pixel, determining a second maximum gray value and a second minimum gray value based on the gray value of the third pixel and the gray values ​​of interpolated pixels adjacent to the third pixel; and determining a first pixel cost corresponding to a specific disparity of the second pixel based on a first difference between the gray value of the second pixel and the second maximum gray value, a second difference between the second minimum gray value and the gray value of the second pixel, a third difference between the gray value of the third pixel and the first maximum gray value, and a fourth difference between the first minimum gray value and the gray value of the third pixel.

[0043] In this way, embodiments of the present disclosure can take into account the continuity of pixels, thereby enabling a more accurate assessment of the degree of difference between pixels.

[0044] In some embodiments of the second aspect, determining the static matching cost between the first image and the second image based on the weighted pixel cost includes: determining an aggregation window associated with the fourth pixel in the first image; and determining the static matching cost of the fourth pixel based on the weighted pixel cost of each pixel within the aggregation window.

[0045] In this way, the embodiments of this disclosure can take into account the influence of differences in surrounding pixels, thereby improving the accuracy of pixel matching.

[0046] In some embodiments of the second aspect, at least one dynamic programming direction includes at least one of the following directions: a first direction indicated by direction vector (1, 0), a second direction indicated by direction vector (1, 1), or a third direction indicated by direction vector (0, 1).

[0047] In this way, the embodiments of this disclosure can take into account multiple dynamic programming directions and facilitate block processing and line-by-line scanning processing of pixels.

[0048] In some embodiments of the second aspect, the processing circuit is further configured to: filter invalid pixels from a row of pixels based on the dynamic matching cost of a row of pixels in the second image block to obtain a set of candidate pixels, wherein the dynamic matching cost of the invalid pixels includes multiple matching costs corresponding to a predetermined disparity range, and the multiple matching costs include at least one interfering matching cost, wherein the difference between the at least one interfering matching cost and the minimum matching cost among the multiple matching costs is less than a predetermined threshold.

[0049] In this way, the embodiments of this disclosure can improve the accuracy of stereo matching through uniqueness detection.

[0050] In some embodiments of the second aspect, the processing circuit is further configured to: determine a set of candidate disparities corresponding to a set of candidate pixels based on a set of optimal matching costs, wherein the optimal matching cost is the minimum matching cost among a plurality of matching costs of candidate pixels corresponding to a predetermined disparity range; and perform left-right consistency detection associated with a set of candidate pixels based on a set of candidate disparities to filter out falsely detected pixels in the set of candidate pixels.

[0051] In this way, the embodiments of this disclosure can improve the accuracy of stereo matching through left-right consistency detection.

[0052] In some embodiments of the second aspect, performing left-right consistency detection associated with a set of candidate pixels includes: determining a set of matching pixels in a second image corresponding to the set of candidate pixels based on a set of candidate disparities; and in response to determining that a target matching pixel in the set of matching pixels corresponds to a plurality of candidate pixels in the set of candidate pixels, determining a false-detection pixel from the plurality of candidate pixels based on a plurality of candidate disparities corresponding to the plurality of candidate pixels, wherein the difference between the candidate disparity corresponding to the false-detection pixel and the target candidate disparity is greater than a predetermined threshold, and the target candidate disparity corresponds to the candidate pixel with the smallest optimal matching cost value between the plurality of candidate pixels and the target matching pixel.

[0053] In this way, embodiments of the present disclosure can perform left-right consistency detection using only one image from the binocular vision images as a reference image, thereby further reducing computational and storage costs.

[0054] In some embodiments of the second aspect, a row of pixels is a first row of pixels, a group of matching pixels is a first group of matching pixels, and the on-chip memory includes a first row buffer and a second row buffer, the first row buffer and the second row buffer being configured to operate in a ping-pong buffer mode, wherein the first row buffer is used to store the disparity corresponding to the first group of matching pixels, and the second row buffer is used to store the disparity corresponding to the second group of matching pixels, the second group of matching pixels being determined based on the second row of pixels in the second image block.

[0055] In this way, embodiments of the present disclosure can utilize a ping-pong buffer mechanism to improve the utilization efficiency of on-chip memory, thereby supporting the parallelization of dynamic cost calculation and left-right consistency detection processes, and effectively improving the efficiency of stereo matching.

[0056] In some embodiments of the second aspect, the first image block and the second image block have overlapping regions, the first image block includes a first non-overlapping region, the second image block includes a second non-overlapping region, and a set of edge pixels includes: a set of pixels in the first non-overlapping region that are adjacent to the overlapping region, or a set of pixels in the overlapping region that are adjacent to the second non-overlapping region.

[0057] In this way, embodiments of the present disclosure improve the accuracy of the post-processing process by partially overlapping the segmented image blocks.

[0058] In some embodiments of the second aspect, a set of edge pixels is a first set of edge pixels, edge cost information is a first edge cost information, and providing a first dynamic matching cost and a second dynamic matching cost includes: storing in on-chip memory a second edge cost information corresponding to the second set of edge pixels in the second dynamic matching cost, wherein the second set of edge pixels includes pixels in the second image block that are adjacent to a third image block among a plurality of image blocks; determining a third dynamic matching cost between the third image block and the second image corresponding to a predetermined disparity range based on the second edge cost information obtained from the on-chip memory; and providing the first dynamic matching cost, the second dynamic matching cost, and the third dynamic matching cost to determine a disparity map between the first image and the second image.

[0059] In this manner, embodiments of the present disclosure cache edge cost information for edge pixels of each intermediate image block to support dynamic matching cost calculation for the next image block. Consequently, embodiments of the present disclosure can reduce the width requirements of on-chip memory, thereby reducing the cost of the processing circuitry.

[0060] In a third aspect of this disclosure, an electronic device is provided. The electronic device includes processing circuitry configured to perform any method of the first aspect and its implementation.

[0061] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions that, when executed by a computing device, cause the computing device to perform any method of the first aspect and its implementation.

[0062] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product includes instructions that, when executed by a computing device, cause the computing device to perform any method of the first aspect and its implementation.

[0063] It is understood that the electronic device of the third aspect, the computer storage medium of the fourth aspect, or the computer program product of the fifth aspect described above can all be used to execute the method provided in the first aspect. Therefore, the explanations or descriptions regarding the first aspect also apply to the third, fourth, and fifth aspects. Furthermore, the beneficial effects achieved by the third, fourth, and fifth aspects can be referred to the beneficial effects in the corresponding methods, and will not be repeated here.

[0064] It should be understood that the description in the Summary of the Invention section is not intended to limit the key or essential features of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0065] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1 A schematic diagram illustrating the basic principle of stereo matching is shown; Figure 2 A schematic diagram is shown illustrating an example environment in which embodiments of the present disclosure can be implemented; Figure 3 A flowchart illustrating an example process of stereo matching according to some embodiments of the present disclosure is shown; Figure 4 A flowchart illustrating an example process for determining a static matching cost according to some embodiments of this disclosure is shown; Figure 5A An example Sobel operator for edge enhancement according to some embodiments of this disclosure is shown; Figure 5B A circuit block diagram illustrating an edge enhancement process according to some embodiments of the present disclosure is shown; Figure 6 A schematic diagram of subpixel interpolation according to some embodiments of the present disclosure is shown; Figure 7 shows a circuit implementation block diagram of the determination point cost matrix according to some embodiments of the present disclosure; Figure 8 A schematic diagram illustrating the aggregation matching cost according to some embodiments of the present disclosure is shown; Figure 9 A circuit implementation block diagram of the aggregate matching cost according to some embodiments of the present disclosure is shown; Figure 10 A schematic diagram of image segmentation according to some embodiments of the present disclosure is shown; Figure 11A and Figure 11BA schematic diagram illustrating the determination of dynamic matching costs according to some embodiments of the present disclosure is shown; Figure 12 A schematic diagram of a left-right consistency detection process according to some embodiments of the present disclosure is shown; Figure 13 A schematic diagram of image segmentation according to yet another embodiment of the present disclosure is shown; Figure 14A and Figure 14B A schematic diagram illustrating the determination of dynamic matching costs according to further embodiments of the present disclosure is shown; and Figure 15 A flowchart illustrating an example method of stereo matching according to some embodiments of the present disclosure is shown.

[0066] In the various figures, the same or similar reference numerals represent the same or similar elements. Detailed Implementation

[0067] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0068] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0069] The basic principle of stereo matching Stereo matching is a technique used to determine disparity between binocular vision images. For example, two cameras can simultaneously capture images, resulting in a left image and a right image. In this case, the difference in coordinates between the same location in space in the left and right images is called disparity. Stereo matching can be used to generate a disparity map between the left and right images.

[0070] Traditionally, stereo matching techniques determine the disparity of each pixel by calculating the matching cost corresponding to different disparities. Figure 1 A schematic diagram 100 illustrates the basic principle of stereo matching. (For example...) Figure 1As shown, the matching cost may differ for different disparities within the disparity search range (also known as the disparity search window). Theoretically, the matching cost for an actual disparity should be the minimum matching cost among all matching costs for all disparities within the disparity search range. Stereo matching techniques can estimate the disparity of a pixel by determining the disparity corresponding to the minimum matching cost within the disparity search window.

[0071] In stereo matching technology, the calculation of the matching cost constitutes the most significant portion of the computational overhead. To improve computational efficiency, specific processing circuits, such as application-specific integrated circuits (ASICs), are designed to accelerate the stereo matching calculation process. Taking a traditional ASIC design for stereo matching as an example, to improve computational efficiency, this type of ASIC design typically integrates a large amount of on-chip memory (e.g., Static Random Access Memory, SRAM) to store parameters related to the matching cost. This results in such ASIC designs requiring on-chip memory with significant area overhead, substantially increasing the cost of the ASIC.

[0072] In view of this, embodiments of the present disclosure provide a scheme for stereo matching. In this scheme, a processing circuit determines a first dynamic matching cost corresponding to a predetermined disparity range between a first image block and a second image, which are formed by a plurality of image blocks of a first image, wherein the first image and the second image form a pair of binocular visual images. Further, the processing circuit stores edge cost information corresponding to a set of edge pixels in the first dynamic matching cost in an on-chip memory, wherein the set of edge pixels includes pixels in the first image block that are adjacent to the second image block among the plurality of image blocks. Accordingly, the processing circuit determines a second dynamic matching cost corresponding to the predetermined disparity range between the second image block and the second image based on the edge cost information obtained from the on-chip memory. The processing circuit determines a disparity map between the first image and the second image based at least on the first dynamic matching cost and the second dynamic matching cost.

[0073] Based on this approach, embodiments of this disclosure can reduce the size of on-chip memory required during the calculation of matching costs by segmenting an image into multiple image blocks and storing the matching costs of edge pixels in on-chip memory, thereby reducing the cost of processing circuitry while maintaining computational efficiency.

[0074] The detailed scheme of this disclosure will now be described with reference to the accompanying drawings.

[0075] Example Environment Figure 2 A schematic diagram of an example environment 200 in which embodiments of the present disclosure can be implemented is shown. (See attached diagram.) Figure 2 As shown, environment 200 may include electronic device 270, which can be used to receive a pair of binocular visual images 250 and 260 to determine a disparity map between the first image 250 and the second image 260.

[0076] like Figure 2 As shown, electronic device 270 may include processing circuitry 210, which may be, for example, an application-specific integrated circuit (ASIC), one or more FPGAs (Field Programmable Gate Arrays), a PLD (Programmable Logic Device), a controller, a state machine, gate logic, discrete hardware components, or any other suitable circuitry.

[0077] like Figure 2 As shown, the processing circuit 210 may include an on-chip memory 220. Compared to the off-chip memory 240, the on-chip memory 220 can have higher access efficiency. However, due to the size and cost limitations of the processing circuit 210, the storage space of the on-chip memory 220 is usually limited. Such an on-chip memory 220 may include, for example, static random access memory (SRAM), magnetic random access memory (MRAM), pseudo static random access memory (PSRAM), etc. In some embodiments, the processing circuit 210 may also include an execution unit (not shown in the figure) for performing operations, which can use the data stored in the on-chip memory 220 to perform a stereo matching process.

[0078] like Figure 2 As shown, the electronic device 270 may also include, for example, an off-chip memory 240 located outside the processing circuitry 210, which has a lower storage cost than the on-chip memory 220, but its access efficiency is also lower than that of the on-chip memory device 220. The processing circuitry 210 may access the off-chip memory 240, for example, via a bus (BUS) 230.

[0079] According to the scheme of using electronic device 270 to achieve stereo matching in this disclosure, processing circuit 210 can divide the first image 250 into multiple image blocks and calculate the dynamic matching cost of each image block separately, which can significantly reduce the size of on-chip storage device 220. In addition, after the dynamic matching cost of the image block is calculated, processing circuit 210 can also store the dynamic matching cost of edge pixels in processing circuit 210 accordingly for use in the dynamic matching cost calculation process of adjacent image blocks.

[0080] The detailed process of performing stereo matching using electronic device 270 will be referenced below. Figures 3 to 12 Detailed description.

[0081] Overall Architecture First refer to Figure 3 , Figure 3 A flowchart illustrating an example process 300 for stereo matching according to some embodiments of the present disclosure is shown. Figure 3 As shown, similar to the traditional stereo matching process, process 300 includes stages such as static matching cost calculation, dynamic programming, post-processing, and disparity map generation.

[0082] First, at 302, processing circuit 210 can calculate the static matching cost. A detailed calculation process for the static matching cost will be provided below. Figures 4 to 9 Detailed description.

[0083] At 304, processing circuit 210 can perform dynamic programming to determine the dynamic matching cost. A detailed explanation of the dynamic programming process will be provided below. Figure 10 , Figure 11A and Figure 11B Detailed description.

[0084] At 306, processing circuit 210 can perform post-processing procedures. Typical post-processing procedures may include uniqueness detection, left-right consistency detection, connectivity detection, etc. It should be understood that, depending on actual needs, processing circuit 210 can perform one or more post-processing procedures, or processing circuit 210 can directly utilize the results of dynamic programming to generate a disparity map. Detailed information about post-processing procedures will be referenced below. Figure 12 describe.

[0085] At 308, processing circuit 210 can generate a disparity map. It should be understood that processing circuit 210 can generate a disparity map based on an estimated disparity determined based on dynamic matching costs, according to an existing appropriate scheme.

[0086] In some embodiments, one or more steps in process 300 may also be performed collaboratively by other computing units in electronic device 270. Electronic device 270 may also include, for example, a central processing unit (CPU) to perform post-processing and disparity map generation based on the dynamic programming results of processing circuit 210.

[0087] Determining the static matching cost Figure 4A flowchart illustrating an example process 302 for determining the static matching cost according to some embodiments of this disclosure is shown. Typically, the static matching cost can be determined using SAD (Sum of Absolute Differences) matching. The calculation principle is as follows: First, a small window is constructed, then the window is used to cover the left side of the image, and all pixels within the covered area are selected. Then, the same window is used to cover the right side of the image, and pixels within the covered area are selected. Finally, the left covered area is subtracted from the right covered area, and the sum of the absolute values ​​of the grayscale differences of all pixels is calculated; this is the SAD matching cost.

[0088] In some embodiments, examples of this disclosure may fuse edge information to improve the accuracy of static matching costs. For example... Figure 4 As shown, at 410, the processing circuit 210 can acquire the left and right images. In some embodiments, the left and right images can be a pair of preprocessed binocular vision images. Such preprocessing may include, but is not limited to, image correction, etc., and this disclosure is not intended to limit the preprocessing process.

[0089] At 420, processing circuit 210 can perform edge enhancement on the left and right images. In some embodiments, processing circuit 210 can utilize Sobel filtering to achieve edge enhancement. Exemplarily, processing circuit 210 can, for example... Figure 5A The Sobel operator 500A is shown in the figure.

[0090] Figure 5B An example circuit implementation of the edge enhancement process is further shown for the 500B. (e.g.) Figure 5B As shown, with 3 The 9 image data corresponding to the 3 matrices (I) 11 to I 33 The values ​​can be obtained and input pairwise as a summer. It should be understood that the summer here performs a subtraction operation to, for example, determine the image data I. 11 and I 13 The difference between them.

[0091] based on Figure 5A The Sobel filter operator shown is I 21 and I 23 The difference can be further input to a value shifter to perform a multiplication by 2 operation. The result can then be fed into an adder and... 11 and I 13 The difference between and I 31 and I 33 The differences between them are summed (the result can be represented as P, for example) to perform further filtering operations.

[0092] In some embodiments, such as Figure 5BAs shown, a threshold PreFilterCap can be defined for filtering, which is summed with the previously determined P to serve as the first selection of the selector. Furthermore, the threshold PreFilterCap can be shifted left by one bit, i.e., multiplied by 2, to serve as the second selection of the selector. Additionally, the third selection of the selector is 0.

[0093] On the other hand, the comparator's judgment result for "P<-PreFilterCap" and "P>PreFilterCap" can be used as the control condition for the selector. When P<-PreFilterCap, the selector can choose the third input, i.e., "0", as the new pixel value, P_new. When P>PreFilterCap, the selector can choose the second input, i.e., "PreFilterCap<<1", as the new pixel value P_new. In other cases, the selector can choose the first input, i.e., "P + PreFilterCap", as the new pixel value P_new.

[0094] Continue to refer to Figure 4 After performing edge enhancement on the left and right images to obtain a pair of enhanced images (e.g., which can be represented as a first enhanced image and a second enhanced image, respectively), the processing circuit 430 can also perform subpixel interpolation on the pair of enhanced images. Figure 6 A schematic diagram 600 illustrating subpixel interpolation according to some embodiments of the present disclosure is shown.

[0095] like Figure 6 As shown, for pixel Il(x) in the left image, processing circuit 210 can insert pixel values ​​Ul and Ur based on its neighboring pixels Il(x-1) and Il(x+1), where Ul=(Il(x-1)+Il(x)) / 2, (1) Ur=(Il(x+1)+Il(x)) / 2. (2) For pixel Ir(xd) in the right image, processing circuit 210 can insert pixel values ​​Vl and Vr based on its neighboring pixels Ir(xd-1) and Ir(x-d+1), where d represents the disparity being searched, and Vl=(Ir(xd-1)+Ir(xd)) / 2, (3) Vr=(Ir(x-d+1)+Ir(xd)) / 2. (4) Continue to refer to Figure 4At 440, processing circuit 210 can calculate the matching cost based on a first interpolated image and a second interpolated image obtained through subpixel interpolation. Unlike directly calculating the difference between two pixels, processing circuit 210 can utilize information from the interpolated pixels to more accurately determine the matching cost of the pixel, also known as the first pixel cost.

[0096] Specifically, the processing circuit 210 can first determine U0, U1, V0, and V1 according to the above formula: U0=min(Ul,Ur,Il(x)) (5) U1=max(Ul,Ur,Il(x)) (6) V0=min(Vl,Vr,Ir(xd)) (7) V1=max(Vl,Vr,Ir(xd)) (8) Furthermore, the processing circuit can determine the first pixel cost pre_Cost(d) of a pixel according to the following formula, where d represents the disparity being searched: pre_C0(d) =max{0,Il(x)-V1,V0-Il(x)} (9) pre_C1(d) =max{0,Ir(xd)-U1,U0-Ir(xd))} (10) pre_Cost(d) = min{pre_C0(d),pre_C1(d)} (11) like Figure 4 As shown, the processing circuit can also perform subpixel interpolation and matching cost calculation on the original image without edge enhancement in steps 450 and 460 respectively, following the process of steps 430 and 440, thereby determining the second pixel cost img_Cost(d) of the pixel. It should be understood that the second pixel cost img_Cost(d) can be determined based on a similar process to formulas (1) to (11), which will not be described in detail here.

[0097] Further, at 470, the processing circuit 210 can determine the weighted pixel cost of a pixel based on the weighted sum of the first pixel cost and the second pixel cost to generate a pixel cost matrix. For example, the processing circuit can determine the weighted pixel cost of each pixel according to the following formula: Cost(d) = a pre_Cost (d) + b img_Cost(d) (12) Where a and b are weighting coefficients. In a specific example, a could be 1 and b could be 0.25.

[0098] Figure 7 further illustrates the circuit implementation block diagram 700 for determining the point cost matrix. As shown in Figure 7, Il1, Il2, and Il3 can represent the pixel information in the original left image, corresponding to Il(x-1), Il(x), and Il(x+1) in formulas (1) and (2), respectively. Similarly, Ir1, Ir2, and Ir3 can represent the pixel information in the original right image, corresponding to Ir(xd-1), Ir(xd), and Ir(x-d+1) in formulas (3) and (4), respectively. Likewise, 'Il1, 'Il2, and 'Il3 correspond to the pixel information in the edge-enhanced left image, and 'Ir1, 'Ir2, and 'Ir3 correspond to the pixel information in the edge-enhanced right image.

[0099] As shown in Figure 7, the operation process described in formulas (1) to (4) can be implemented by an adder and a shifter, and the operation process described in formulas (5) to (8) can be implemented by a comparator. Furthermore, the calculation process described in formulas (9) to (11) can be implemented by combining an adder and a comparator to obtain img_Cost(d) and pre_Cost(d) respectively.

[0100] As shown in Figure 7, a shifter can be used to shift img_Cost(d) two bits to the right and sum it with pre_Cost(d) to obtain the final Cost(d), thus realizing the weighted summation calculation process described in formula (12). Based on this method, the point cost matrix of each pixel in the left image can be determined (which can be represented as pixDiff).

[0101] Furthermore, at 480, the processing circuit can perform window aggregation on the point cost matrix to generate a static matching cost. Figure 8 A schematic diagram 800 illustrates the aggregation matching cost according to some embodiments of the present disclosure. For example... Figure 8 As shown, in 1920 Using 1080 as an example for creating an image, for pixel p, we can determine 9 A 9-pixel aggregation window is defined, and the sum of the weighted pixel costs of the 81 pixels within this aggregation window is determined as the static matching cost of pixel p, which can be expressed as: (13) Where x and y represent the coordinates of pixel p, d represents the disparity being searched, and n represents the size of the aggregation window.

[0102] Figure 9 A circuit implementation block diagram 900 of the aggregate matching cost according to some embodiments of the present disclosure is further shown. Figure 9As shown, multiple adders can be used to first calculate the cost sum of each row, and then the cost sum of each row can be accumulated to determine the SAD within the aggregation window, which is used as the static matching cost of pixel p.

[0103] The above describes the process for determining the static matching cost according to this disclosure. By fusing information from the edge enhancement image and the original image, the static matching cost can be determined more accurately.

[0104] Determining the cost of dynamic matching The following will be referenced Figure 10 , Figure 11A and Figure 11B This describes the process of determining the dynamic matching cost according to embodiments of the present disclosure. Figure 10 A schematic diagram 1000 illustrating image segmentation according to some embodiments of the present disclosure is shown. For example... Figure 10 As shown, for an input pair of binocular visual images (first image 250 and second image 260), the electronic device 270 can first divide the first image 250 into multiple image blocks, for example, image blocks 1010-1, 1010-2, 1010-3, and 1010-4 (each individually or collectively referred to as image block 1010). In some embodiments, such as Figure 10 As shown, multiple image patches 1010 can be adjacent to each other without overlapping areas. This partitioning method improves computational efficiency. It should be understood that... Figure 10 The specific number and size of the image blocks shown are merely illustrative.

[0105] Accordingly, the processing circuit 210 can determine the image regions in the second image 260 that match the image blocks in the first image 250 based on the disparity range to be searched (e.g., 0 to d). For example, image block 1010-1 can correspond to image block 1020-1, image block 1010-2 can correspond to image block 1020-2 and a pixel region of width d in image block 1020-1, image block 1010-3 can correspond to image block 1020-3 and a pixel region of width d in image block 1020-2, and image block 1010-4 can correspond to image block 1020-4 and a pixel region of width d in image block 1020-3. In this way, the size of the required line buffer can be reduced to one-quarter of that in previous schemes during each line processing, thereby reducing the cost of on-chip memory.

[0106] Traditionally, the dynamic matching cost can be determined iteratively based on the static cost per pixel. (14) in, Lr (p , d ) represents the dynamic matching cost of pixel p in the left image, in the dynamic programming direction indicated by r, corresponding to the disparity d; C ( p , d ) represents the static matching cost of pixel p in the left image corresponding to disparity d, which can be determined, for example, according to the process discussed above or other appropriate methods; r represents the direction vector of the dynamic programming direction; P1 and P2 are constants representing the multiplication coefficients.

[0107] In some embodiments, to facilitate block-based dynamic computation, the dynamic programming direction may include one or more of the following: a first direction indicated by direction vector (1, 0), a second direction indicated by direction vector (1, 1), and a third direction indicated by direction vector (0, 1).

[0108] It should be understood that directions indicated by other direction vectors may also be included. For the convenience of block-based and row-scan calculations, such directions may indicate one or more angles, for example, ranging from 90 degrees downwards to 90 degrees to the right.

[0109] Furthermore, the processing device 210 can determine the dynamic matching cost of the pixel based on the sum of the dynamic matching costs in multiple dynamic programming directions, that is: (15) Typically, the dynamic matching cost of each pixel can be determined iteratively according to formulas (14) and (15). However, this would require a larger on-chip memory to store the intermediate data generated by formulas (14) and (15).

[0110] Figure 11A and Figure 11B Schematic diagrams 1100A and 1100B are shown illustrating the determination of dynamic matching costs according to some embodiments of the present disclosure. Figure 11A As shown, for the first image block 1010-1 among multiple image blocks, the processing circuit 210 can determine the dynamic matching cost of each pixel in the first image block in a conventional manner according to formulas (14) and (15) and the determined static matching cost.

[0111] like Figure 11AAs shown, for pixel 1110, processing circuit 210 can iteratively calculate the dynamic matching cost in three dynamic programming directions. For example, the dynamic matching cost in the first direction can be calculated based on the dynamic matching cost of pixel S0; correspondingly, the dynamic matching cost of S0 can be iteratively converted into the dynamic matching cost of the left pixel in the first direction. In this way, processing circuit 210 can iteratively calculate the dynamic matching cost of the pixel in the first direction. The calculation of the dynamic matching cost in the other two directions can be handled similarly. In such an iterative process, other pixels in the dynamic programming directions can also be referred to as the associated pixels of the pixel.

[0112] Unlike traditional methods, such as Figure 11B As shown, the processing circuit 210 can store the edge cost information of a group of edge pixels in the first image block 1010-1 into the on-chip memory 220. The edge cost information of this group of edge pixels includes the dynamic matching cost of the group of edge pixels in at least one dynamic programming direction. Here, edge pixels refer to pixels in the first image block 1010-1 that are adjacent to the second image block 1010-2, such as pixel 1110.

[0113] Furthermore, the processing circuit 210 can obtain edge cost information of edge pixels from the on-chip memory 220 and use the edge cost information to determine the dynamic matching cost of each pixel in the second image block 1010-2. Taking pixel 1120 as an example, the dynamic matching cost of pixel 1120 in the first direction can be determined iteratively based on the dynamic matching cost of pixel 1110 in the first direction. Based on this method, the processing circuit 210 does not need to use the dynamic matching costs of other pixels of non-edge pixels in the first image block 1010-1, thereby effectively reducing the size requirement of the on-chip memory 220 and thus reducing the cost of the processing circuit 210.

[0114] It should be understood that the processing circuit 210 can store the edge cost information of the edge pixels in the previous image block into the on-chip memory 220 in a similar manner, and use it to determine the dynamic matching cost of the next image block, thereby completing the determination of the dynamic matching cost of all multiple image blocks.

[0115] For example, the edge cost information of a group of edge pixels in the second image block 1010-2 that are adjacent to the third image block 1010-3 can be stored in the on-chip memory 220 to determine the dynamic matching cost of each pixel in the third image block 1010-3. Similarly, the edge cost information of a group of edge pixels in the third image block 1010-3 that are adjacent to the fourth image block 1010-4 can be stored in the on-chip memory 220 to determine the dynamic matching cost of each pixel in the fourth image block 1010-4. In this way, the dynamic matching cost of all pixels in the first image 250 can be determined to determine the disparity map.

[0116] Post-processing After determining the dynamic matching cost of pixels, post-processing can be performed to improve matching accuracy. (See reference above.) Figure 3 The post-processing process discussed may be performed by the processing circuit 210, or by other suitable processing units in the electronic device 270. The post-processing process will be described below using the processing circuit 210 as an example.

[0117] To improve the utilization efficiency of the on-chip memory 220, the processing circuit 210 can determine the dynamic cost by processing pixels row by row, and can perform post-processing immediately after determining one row of pixels in each image block. In some embodiments, the post-processing may include uniqueness detection. Specifically, taking the second image block 1010-2 as an example, the processing circuit 210 can first determine a set of dynamic matching costs corresponding to one row of pixels in the second image block 1010-2 and the disparity search range (e.g., 0-127) according to the above process. Further, the processing circuit 210 can filter out invalid pixels based on the uniqueness detection mechanism. Specifically, if at least one of the multiple dynamic matching costs (e.g., corresponding to disparity 0-127) corresponding to a specific pixel has a difference of less than a predetermined threshold from the minimum dynamic matching cost, then the specific pixel can be determined as an invalid pixel. For example, if a particular pixel has the minimum dynamic matching cost cost1 at disparity "10" and the dynamic matching cost corresponding to disparity "20" is cost2, then if the difference between cost1 and cost2 is less than a predetermined threshold, then that particular pixel can be identified as an invalid pixel.

[0118] It should be understood that the difference between dynamic matching costs can be represented by a difference or a ratio. In this way, situations where multiple pixels might match can be further filtered out. In this disclosure, multiple pixels that pass the uniqueness detection can also be referred to as candidate pixels.

[0119] Considering that in some cases (such as occlusion), some pixels in the left image may match multiple pixels in the right image, the post-processing may further include left-right consistency detection in some embodiments. In some embodiments, the processing circuit 210 may perform left-right consistency detection line by line on the disparity. In some embodiments, the processing circuit 210 may perform left-right consistency detection using the dynamic matching cost stored in the on-chip memory 220 after determining the dynamic matching cost of a row of pixels. In some embodiments, the pixels targeted for left-right consistency detection may include only candidate pixels that have passed the uniqueness detection.

[0120] Specifically, the processing circuit 210 can determine a set of optimal matching costs for a group of candidate pixels obtained after a row of pixels passes consistency detection. The optimal matching cost refers to the minimum matching cost among multiple matching costs corresponding to a candidate pixel and a predetermined disparity range. Further, the processing circuit 210 can determine a set of candidate disparities corresponding to a group of candidate pixels based on a set of optimal matching costs for a group of candidate pixels. For example, the processing circuit 210 can determine that the candidate disparity corresponding to pixel (500, 1) is, for example, 7, based on the 128 dynamic matching costs corresponding to pixel (500, 1) and the disparity range (0-127). This indicates that pixel (500, 1) corresponds to pixel (493, 1) in the right image.

[0121] Furthermore, the processing circuit 210 can perform left-right consistency detection associated with the group of candidate pixels based on the group of candidate disparities to filter out falsely detected pixels in the group of candidate pixels.

[0122] Traditional left-right consistency detection can determine whether left-right consistency is satisfied based on whether the difference between the disparity value d1 of a pixel determined with the left image as a reference image (e.g., coordinate x) and the disparity value of the corresponding pixel determined with the right image as a reference image (i.e., coordinate x-d1) is greater than a threshold.

[0123] In some embodiments, the processing circuit 210 may also perform left-right consistency detection based solely on the disparity information determined by using one of the images as a reference image. Specifically, the processing circuit 210 may determine a set of candidate pixels in the row of pixels according to the uniqueness detection process described above. Alternatively, the processing circuit 210 may directly use the row of pixels as a set of candidate pixels.

[0124] Furthermore, the processing circuit 210 can determine a set of optimal matching costs for the group of candidate pixels. The optimal matching cost is the minimum matching cost among multiple matching costs corresponding to a predetermined disparity range for the candidate pixel. For example, for each candidate pixel, the processing circuit 210 can determine the minimum dynamic matching cost corresponding to it within the disparity range as the optimal matching cost.

[0125] Additionally, the processing circuit 210 can also determine a set of candidate disparities corresponding to the set of candidate pixels based on a set of optimal matching costs. For example, the processing circuit 210 can utilize on-chip memory 220 to store the candidate disparities corresponding to each candidate pixel, where the candidate disparity refers to the disparity corresponding to the optimal matching cost. Further, the processing circuit 210 can perform left-right consistency detection based on the set of candidate disparities to filter out falsely detected pixels in the set of candidate pixels.

[0126] Specifically, the processing circuit 210 can determine a set of matching pixels in the second image 260 corresponding to the set of candidate pixels based on the candidate disparity. For example, if a candidate pixel is pixel (500, 1) in the first image 250, and its determined candidate disparity is "20", then the matching pixel in the second image 260 corresponding to that pixel is (480, 1). However, as discussed above, due to occlusion or other reasons, multiple pixels may correspond to the same matching pixel. For example, if the candidate disparity corresponding to pixel (520, 1) in the first image 250 is "40", then it also corresponds to the matching pixel (480, 1). This may lead to mismatch problems.

[0127] Furthermore, if it is determined that the target matching pixel in the set of matching pixels corresponds to multiple candidate pixels in the set of candidate pixels, the processing circuit 210 can determine the false detection pixel from the multiple candidate pixels based on the multiple candidate disparities corresponding to the multiple candidate pixels, wherein the difference between the candidate disparity corresponding to the false detection pixel and the target candidate disparity is greater than a predetermined threshold, and the target candidate disparity corresponds to the candidate pixel with the smallest optimal matching cost value with the target matching pixel among the multiple candidate pixels.

[0128] Continuing the previous example, if the matching pixel (480, 1) corresponds to both pixel (500, 1) and pixel (520, 1) in the first image 250, then the processing circuit 210 can determine the pixel with the minimum dynamic matching cost from pixels (500, 1) and (520, 1), for example, (520, 1). Further, the processing circuit 210 can determine the disparity "40" corresponding to pixel (520, 1) as a target candidate disparity.

[0129] For other candidate pixels, the processing circuit 210 can compare its corresponding candidate disparity with the target candidate disparity. If the difference is greater than a predetermined threshold, the processing circuit 210 can identify the candidate pixel as a false detection pixel. For example, for pixel (500, 1), its corresponding disparity is "20". If the difference "20" between it and the target candidate disparity "40" is greater than the predetermined threshold, the processing circuit 210 can identify it as a false detection pixel. Conversely, if there is another pixel (505, 1), which also corresponds to the matching pixel (480, 1), and the difference "5" between its candidate disparity "45" and the target disparity "40" is less than the predetermined threshold, the processing circuit 210 can consider pixel (505, 1) to also correspond to the matching pixel (480, 1). Based on this method, the embodiments of this disclosure can improve the accuracy of stereo matching.

[0130] In terms of specific circuit implementation... Figure 12 A schematic diagram 1200 further illustrates a left-right consistency detection process according to some embodiments of the present disclosure. For example... Figure 12 As shown, S(pd) can represent the dynamic matching cost of a row of pixels in the second image block corresponding to a predetermined disparity range, as determined by the dynamic matching cost determination process described above.

[0131] Furthermore, such as Figure 12 As shown, the processing circuit 210 can, for example, determine a set of matching pixels corresponding to the first row of pixels based on S(pd), and use a line buffer (referred to as the first line buffer for convenience) to store the disparity corresponding to the set of matching pixels. Specifically, the processing circuit 210 can, for example, process multiple candidate pixels sequentially according to coordinate order, and store disparity information in the first line buffer at the positions corresponding to the matching pixels of each candidate pixel.

[0132] For example, if a candidate pixel is located at (x, y) and its candidate disparity is d, the processing circuit 210 can use the candidate disparity d to update the disparity stored at position (xd) in the first row buffer. In some embodiments, the processing circuit 210 can determine whether a disparity is stored at position (xd). If not, the processing circuit 210 can directly write the disparity d to that position. Conversely, if a disparity has already been written at position (xd), the processing circuit 210 can determine whether the dynamic matching cost corresponding to the disparity d to be written is less than the dynamic matching cost corresponding to the previously written disparity, and only if it is less will the disparity d be written to the first row buffer. Based on this method, the first row buffer can store the disparity corresponding to a set of matching pixels.

[0133] Furthermore, the processing circuit 210 can utilize the disparity corresponding to this set of matching pixels to perform left-right consistency detection. Specifically, as... Figure 12 As shown, for each candidate pixel, the processing circuit 210 can perform sub-pixel interpolation based on its candidate disparity, and then interpolate the disparity after sub-pixel interpolation (such as...). Figure 12 The best_Disparity shown is written to another buffer.

[0134] During the left-right consistency detection process, the processing circuit 210 can determine whether the following formula holds true: abs(disp2ptr[_x]-_d)>disp12MaxDiff&&abs(disp2ptr[x_]-d_)>disp12MaxDiff (16) Where _d represents the floor function of d, d_ represents the floor function of d, abs represents the absolute value, disp2ptr[] represents the disparity stored in the first row buffer, _x represents the pixel in the right image corresponding to _d, and x_ represents the pixel in the right image corresponding to d_. If formula (16) holds, the target pixel is considered to have failed the left-right consistency detection and can be identified as a falsely detected pixel.

[0135] In some embodiments, to improve the efficiency of the processing circuit, the processing circuit 210 may, for example, utilize two row buffers to process the left-right consistency detection of different rows separately. Specifically, as... Figure 12 As shown, the processing circuit 210 can also utilize another row buffer (referred to as the second row buffer for convenience) to similarly store the disparity corresponding to the second set of matching pixels, wherein the second set of matching pixels is determined based on the second row of pixels in the second image block. It should be understood that the terms "first row of pixels" and "second row of pixels" here are intended to indicate that they represent different rows of pixels in the second image block, and are not intended to limit the specific location of the pixel row in the second image block.

[0136] In some embodiments, the first row buffer and the second row buffer can operate in a ping-pong buffer mode. Also known as a dual buffer, in this mode, while one buffer is writing data, the other buffer can simultaneously perform data processing, thereby improving processing efficiency.

[0137] Specifically, in a stereo matching scenario, the processing circuit 210 can, for example, first perform the dynamic matching cost calculation for one row of pixels in the second image block, and then use the first row buffer to perform left-right consistency detection for that row of pixels. While performing the left-right consistency detection, the processing circuit 210 can use the second row buffer to perform the dynamic matching cost calculation for the next row of pixels. Based on this approach, the processing circuit 210 can parallelize the dynamic matching cost calculation and post-processing, thereby improving processing efficiency.

[0138] After filtering the pixels that have passed the left-right consistency test, the electronic device 270 can further determine the disparity map between the first image 250 and the second image 260 based on the optimal disparity of these pixels, thereby achieving stereo matching between the two. It should be understood that any suitable method can be used to construct the disparity map, which will not be detailed herein.

[0139] In some embodiments, if there is no overlapping area between image blocks during the block processing, the post-processing is also performed based on one row of pixels in each image block, which may lead to a decrease in the accuracy of some post-processing processes. For example, during the left-right consistency detection process, the processing circuit 210 may be unable to determine if pixels in the first image block 1010-1 and the second image block 1010-2 match the same pixel in the second image 260.

[0140] In some embodiments, to improve the accuracy of the post-processing, the first image 250 may be divided into image blocks with overlapping regions, for example. Figure 13 A schematic diagram 1300 illustrating image segmentation according to yet another embodiment of the present disclosure is shown. For example... Figure 13 As shown, the first image 250 can, for example, be divided into four image blocks 1310-1, 1310-2, 1310-3, and 1310-4 (each individually or collectively referred to as image block 1310). Figure 10 Different partitioning schemes are shown, and image blocks 1310 may have overlapping areas. For example... Figure 13 As shown, image block 1310-1 and image block 1310-2 may have an overlapping region 1320. In some embodiments, the width of the overlapping region 1320 may correspond to, for example, the maximum search parallax.

[0141] It should be understood that, in conjunction with the above reference Figure 11A and Figure 11B The process of determining the dynamic matching cost is similar to that described above. The processing circuit 210 can also use the on-chip memory 220 to store the edge cost information of the edge pixels. Figure 14A and Figure 14BSchematic diagrams 1400A and 1400B are shown illustrating the determination of dynamic matching costs according to further embodiments of the present disclosure.

[0142] In some embodiments, the processing circuit 210 may, for example, determine the dynamic matching cost of each pixel in the first image block 1310-1 by referring to the method for determining dynamic matching costs described above. Further, the processing circuit 210 may, for example, utilize the on-chip memory 220 to store edge cost information of edge pixels, which may include, for example, dynamic matching costs in multiple dynamic programming directions.

[0143] In some embodiments, as shown in FIG1400A, edge pixels may include, for example, pixels 1410 in the first image block 1310-1 that are adjacent to the first non-overlapping region and the overlapping region 1320. In this way, the processing circuit 210 can use the edge cost information of the pixels 1410 stored in the on-chip memory 220 to determine the dynamic matching cost of each pixel 1420 (including pixels in the overlapping region 1320) in the second image block 1310-2.

[0144] In some embodiments, as shown in FIG1400B, edge pixels may include, for example, pixels 1430 adjacent to non-overlapping regions in the overlapping region 1320 and in the second image block 1310-2. Based on this, the processing circuit 210 can use the edge cost information of pixel 1430 stored in the on-chip memory 220 to determine the dynamic matching cost of each pixel 1440 (including pixels in the overlapping region 1320) in the second image block 1310-2.

[0145] In some embodiments, if it is necessary to utilize the dynamic matching cost of pixels stored in the on-chip memory 220 of the processing circuit 210 to perform post-processing, then for Figure 14B In the scheme shown, the on-chip memory 220 may, for example, store the dynamic matching cost of all pixels in the overlapping region 1320 determined by the processing circuit 210 when processing the first image block 1310-1, so as to facilitate the post-processing of the second image block 1310-2.

[0146] It should be understood that post-processing can be performed by referring to the process discussed above, and... Figure 10 The different partitioning schemes shown indicate that for Figure 13 The partitioning scheme shown allows the processing circuit 210 to improve the inspection range of uniqueness detection and / or left-right consistency detection during post-processing because there are overlapping areas between different image blocks 1310, thereby improving the accuracy of the post-processing process.

[0147] Taking left-right consistency detection as an example, for Figure 13The partitioning scheme shown allows the width range of the first image block 1310-1 to be, for example, [0, 543], and the width range of the second image block 1310-2 to be, for example, [415, 895], with a disparity search range of, for example, [0, 127]. Therefore, the maximum distance between two falsely detected pixels is 128. Consequently, the processing circuit 210 can always eliminate falsely detected pixels during the left-right consistency detection process corresponding to either the first image block 1310-1 or the second image block 1310-2. Thus, embodiments of this disclosure can improve the accuracy of the post-processing.

[0148] Example Method Figure 15 A flowchart of a stereo matching method 1500 according to an embodiment of the present disclosure is shown. Method 1500 may, for example, be derived from... Figure 2 The processing circuit 210 is implemented in the middle.

[0149] like Figure 15 As shown, in block 1502, the processing circuit 210 determines a first dynamic matching cost between a first image block and a second image, which corresponds to a predetermined disparity range, and the first image and the second image form a pair of binocular visual images.

[0150] In block 1504, the processing circuit 210 stores edge cost information corresponding to a set of edge pixels in the first dynamic matching cost in on-chip memory, wherein the set of edge pixels includes pixels in the first image block that are adjacent to a second image block among a plurality of image blocks.

[0151] In block 1506, processing circuit 210 determines a second dynamic matching cost between the second image block and the second image corresponding to a predetermined disparity range based on edge cost information obtained from on-chip memory.

[0152] In block 1508, processing circuitry 210 provides a first dynamic matching cost and a second dynamic matching cost for determining a disparity map between the first image and the second image.

[0153] In this way, embodiments of the present disclosure can reduce the size of on-chip memory required in calculating the matching cost by dividing the image into multiple image blocks and storing the matching cost of edge pixels in on-chip memory, thereby reducing the cost of processing circuitry while maintaining computational efficiency.

[0154] In some embodiments of the first aspect, determining the first dynamic matching cost includes: determining a static matching cost between the first image patch and the second image corresponding to a predetermined disparity range; and determining a dynamic matching cost of the first pixel based on the static matching cost of the first pixel in the first image patch and the dynamic matching cost of at least one associated pixel associated with the first pixel, wherein the at least one associated pixel is determined based on at least one dynamic programming direction.

[0155] In this way, embodiments of the present disclosure can determine the dynamic matching cost between pixels based on the static matching cost, thereby improving the accuracy of matching.

[0156] In some embodiments of the first aspect, the edge cost information includes the dynamic matching cost of the edge pixel associated with at least one dynamic programming direction.

[0157] In this way, embodiments of the present disclosure can store the dynamic matching cost of edge pixels in at least one planning direction in on-chip memory, thereby improving the basic data for the calculation of dynamic cost in the planning direction in subsequent image blocks, thereby reducing the storage requirements of on-chip memory.

[0158] In some embodiments of the first aspect, determining the static matching cost includes: performing edge enhancement on a first image and a second image respectively to obtain a first enhanced image and a second enhanced image; determining a first pixel cost associated with a plurality of pixels in the first image based on the first enhanced image and the second enhanced image; determining a second pixel cost of the plurality of pixels based on the first image and the second image; determining a weighted pixel cost of the plurality of pixels based on a weighted sum of the first pixel cost and the second pixel cost; and determining a static matching cost between the first image and the second image based on the weighted pixel cost.

[0159] In this way, embodiments of the present disclosure can more accurately determine the static matching cost between pixels by utilizing a combination of the static matching costs of the edge-enhanced image and the original image.

[0160] In some embodiments of the first aspect, determining the first pixel cost includes: performing subpixel interpolation on the first enhanced image and the second enhanced image respectively to obtain a first interpolated image and a second interpolated image; and determining the first pixel cost associated with a plurality of pixels in the first image based on the first interpolated image and the second interpolated image.

[0161] In this way, embodiments of the present disclosure can more accurately assess the degree of difference between pixels through subpixel interpolation.

[0162] In some embodiments of the first aspect, determining a first pixel cost associated with a plurality of pixels in the first image based on a first interpolated image and a second interpolated image includes: for a second pixel among the plurality of pixels, determining a first maximum gray value and a first minimum gray value based on the gray value of the second pixel and the gray values ​​of interpolated pixels adjacent to the second pixel; for a third pixel in the second image that has a specific disparity with the second pixel, determining a second maximum gray value and a second minimum gray value based on the gray value of the third pixel and the gray values ​​of interpolated pixels adjacent to the third pixel; and determining a first pixel cost corresponding to a specific disparity of the second pixel based on a first difference between the gray value of the second pixel and the second maximum gray value, a second difference between the second minimum gray value and the gray value of the second pixel, a third difference between the gray value of the third pixel and the first maximum gray value, and a fourth difference between the first minimum gray value and the gray value of the third pixel.

[0163] In this way, embodiments of the present disclosure can take into account the continuity of pixels, thereby enabling a more accurate assessment of the degree of difference between pixels.

[0164] In some embodiments of the first aspect, determining the static matching cost between the first image and the second image based on the weighted pixel cost includes: determining an aggregation window associated with the fourth pixel in the first image; and determining the static matching cost of the fourth pixel based on the weighted pixel cost of each pixel within the aggregation window.

[0165] In this way, the embodiments of this disclosure can take into account the influence of differences in surrounding pixels, thereby improving the accuracy of pixel matching.

[0166] In some embodiments of the first aspect, at least one dynamic programming direction includes at least one of the following directions: a first direction indicated by direction vector (1, 0), a second direction indicated by direction vector (1, 1), or a third direction indicated by direction vector (0, 1).

[0167] In this way, the embodiments of this disclosure can take into account multiple dynamic programming directions and facilitate block processing and line-by-line scanning processing of pixels.

[0168] In some embodiments of the first aspect, the method further includes: filtering invalid pixels from a row of pixels based on a dynamic matching cost of a row of pixels in a second image block to obtain a set of candidate pixels, wherein the dynamic matching cost of the invalid pixels includes multiple matching costs corresponding to a predetermined disparity range, and the multiple matching costs include at least one interfering matching cost, wherein the difference between the at least one interfering matching cost and the minimum matching cost among the multiple matching costs is less than a predetermined threshold.

[0169] In this way, the embodiments of this disclosure can improve the accuracy of stereo matching through uniqueness detection.

[0170] In some embodiments of the first aspect, the method further includes: determining a set of candidate disparities corresponding to a set of candidate pixels based on a set of optimal matching costs, wherein the optimal matching cost is the minimum matching cost among a plurality of matching costs corresponding to a predetermined disparity range for the candidate pixels; and performing left-right consistency detection associated with the set of candidate pixels based on the set of candidate disparities to filter out falsely detected pixels in the set of candidate pixels.

[0171] In this way, the embodiments of this disclosure can improve the accuracy of stereo matching through left-right consistency detection.

[0172] In some embodiments of the first aspect, performing left-right consistency detection associated with a set of candidate pixels includes: determining a set of matching pixels in a second image corresponding to the set of candidate pixels based on a set of candidate disparities; and in response to determining that a target matching pixel in the set of matching pixels corresponds to a plurality of candidate pixels in the set of candidate pixels, determining a false-detection pixel from the plurality of candidate pixels based on a plurality of candidate disparities corresponding to the plurality of candidate pixels, wherein the difference between the candidate disparity corresponding to the false-detection pixel and the target candidate disparity is greater than a predetermined threshold, and the target candidate disparity corresponds to the candidate pixel with the smallest optimal matching cost value between the plurality of candidate pixels and the target matching pixel.

[0173] In this way, embodiments of the present disclosure can perform left-right consistency detection using only one image from the binocular vision images as a reference image, thereby further reducing computational and storage costs.

[0174] In some embodiments of the first aspect, a row of pixels is a first row of pixels, a set of matching pixels is a first set of matching pixels, and the on-chip memory includes a first row buffer and a second row buffer, the first row buffer and the second row buffer being configured to operate in a ping-pong buffer mode, wherein the first row buffer is used to store the disparity corresponding to the first set of matching pixels, and the second row buffer is used to store the disparity corresponding to the second set of matching pixels, the second set of matching pixels being determined based on the second row of pixels in the second image block.

[0175] In this way, embodiments of the present disclosure can utilize a ping-pong buffer mechanism to improve the utilization efficiency of on-chip memory, thereby supporting the parallelization of dynamic cost calculation and left-right consistency detection processes, and effectively improving the efficiency of stereo matching.

[0176] In some embodiments of the first aspect, the first image block and the second image block have overlapping regions, the first image block includes a first non-overlapping region, the second image block includes a second non-overlapping region, and a set of edge pixels includes: a set of pixels in the first non-overlapping region that are adjacent to the overlapping region, or a set of pixels in the overlapping region that are adjacent to the second non-overlapping region.

[0177] In this way, embodiments of the present disclosure improve the accuracy of the post-processing by partially overlapping the segmented image blocks.

[0178] In some embodiments of the first aspect, a set of edge pixels is a first set of edge pixels, edge cost information is a first edge cost information, and providing a first dynamic matching cost and a second dynamic matching cost includes: storing in an on-chip memory a second edge cost information corresponding to a second set of edge pixels in the second dynamic matching cost, wherein the second set of edge pixels includes pixels in a second image block that are adjacent to a third image block among a plurality of image blocks; determining a third dynamic matching cost between the third image block and the second image corresponding to a predetermined disparity range based on the second edge cost information obtained from the on-chip memory; and providing the first dynamic matching cost, the second dynamic matching cost, and the third dynamic matching cost for determining a disparity map between the first image and the second image.

[0179] In this manner, embodiments of the present disclosure cache edge cost information for edge pixels of each intermediate image block to support dynamic matching cost calculation for the next image block. Consequently, embodiments of the present disclosure can reduce the width requirements of on-chip memory, thereby reducing the cost of the processing circuitry.

[0180] In some embodiments, this disclosure also provides an electronic device including the processing circuitry 210 discussed in this disclosure. The electronic device can utilize the processing circuitry 210 to perform a stereo matching process discussed in this disclosure.

[0181] This disclosure can be a method, processing circuit, electronic device, computer storage medium, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0182] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0183] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0184] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0185] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0186] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0187] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0188] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0189] Various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the disclosed embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for stereo matching, the method being executed by a processing circuit, the processing circuit including an on-chip memory, characterized in that, The method includes: The processing circuit determines a first dynamic matching cost corresponding to a predetermined disparity range between a first image block in a plurality of image blocks in a first image and an image region in a second image corresponding to the first image block, and the first image and the second image form a pair of binocular visual images; The on-chip memory stores edge cost information corresponding to a set of edge pixels in the first dynamic matching cost, wherein the set of edge pixels includes pixels in the first image block that are adjacent to a second image block among the plurality of image blocks; Based on the edge cost information obtained from the on-chip memory, a second dynamic matching cost corresponding to the predetermined disparity range is determined between the second image block and the image region in the second image corresponding to the second image block; and The first dynamic matching cost and the second dynamic matching cost are provided to determine the disparity map between the first image and the second image.

2. The method of claim 1, wherein determining the first dynamic matching cost includes: Determine the static matching cost corresponding to the predetermined disparity range between the first image block and the image region in the second image corresponding to the first image block; as well as The dynamic matching cost of the first pixel is determined based on the static matching cost of the first pixel in the first image block and the dynamic matching cost of at least one associated pixel associated with the first pixel, wherein the at least one associated pixel is determined based on at least one dynamic programming direction.

3. The method of claim 2, wherein the edge cost information includes the dynamic matching cost of the edge pixel associated with the at least one dynamic programming direction.

4. The method of claim 2, wherein determining the static matching cost comprises: Edge enhancement is performed on the first image and the second image respectively to obtain a first enhanced image and a second enhanced image; Based on the first enhanced image and the second enhanced image, a first pixel cost associated with multiple pixels in the first image is determined; Based on the first image and the second image, determine the second pixel cost of the plurality of pixels; The weighted pixel cost of the plurality of pixels is determined based on the weighted sum of the first pixel cost and the second pixel cost. as well as Based on the weighted pixel cost, the static matching cost between the first image and the second image is determined.

5. The method of claim 4, wherein determining the first pixel cost comprises: Subpixel interpolation is performed on the first enhanced image and the second enhanced image respectively to obtain a first interpolated image and a second interpolated image; as well as Based on the first interpolated image and the second interpolated image, the first pixel cost associated with the plurality of pixels in the first image is determined.

6. The method of claim 5, wherein determining the first pixel cost associated with the plurality of pixels in the first image based on the first interpolated image and the second interpolated image comprises: For the second pixel among the plurality of pixels, a first maximum gray value and a first minimum gray value are determined based on the gray value of the second pixel and the gray value of the interpolated pixels adjacent to the second pixel. For a third pixel in the second image that has a specific disparity with the second pixel, a second maximum gray value and a second minimum gray value are determined based on the gray value of the third pixel and the gray values ​​of the interpolated pixels adjacent to the third pixel; and Based on the first difference between the gray value of the second pixel and the second maximum gray value, the second difference between the second minimum gray value and the gray value of the second pixel, the third difference between the gray value of the third pixel and the first maximum gray value, and the fourth difference between the first minimum gray value and the gray value of the third pixel, the first pixel cost corresponding to the specific disparity of the second pixel is determined.

7. The method of claim 4, wherein determining the static matching cost between the first image and the second image based on the weighted pixel cost comprises: Determine the aggregation window associated with the fourth pixel in the first image; as well as The static matching cost of the fourth pixel is determined based on the weighted pixel cost of each pixel within the aggregation window.

8. The method of claim 2, wherein the at least one dynamic programming direction includes at least one of the following directions: The first direction indicated by the direction vector (1, 0), The second direction indicated by the direction vector (1,1), or The third direction indicated by the direction vector (0, 1).

9. The method according to claim 1, further comprising: Based on the dynamic matching cost of a row of pixels in the second image block corresponding to the predetermined disparity range, invalid pixels are filtered from the row of pixels to obtain a set of candidate pixels. The dynamic matching cost of the invalid pixels includes multiple matching costs corresponding to the predetermined disparity range, and the multiple matching costs include at least one interference matching cost. The difference between the at least one interference matching cost and the minimum matching cost among the multiple matching costs is less than a predetermined threshold.

10. The method of claim 9, further comprising: Based on a set of optimal matching costs for the set of candidate pixels, a set of candidate disparities corresponding to the set of candidate pixels is determined, wherein the optimal matching cost is the minimum matching cost among multiple matching costs of the candidate pixels corresponding to the predetermined disparity range. as well as Based on the set of candidate disparities, perform left-right consistency detection associated with the set of candidate pixels to filter out falsely detected pixels in the set of candidate pixels.

11. The method of claim 10, wherein performing left-right consistency detection associated with the set of candidate pixels comprises: Based on the set of candidate disparities, a set of matching pixels in the second image corresponding to the set of candidate pixels is determined; as well as In response to determining that the target matching pixel in the set of matching pixels corresponds to multiple candidate pixels in the set of candidate pixels, a false detection pixel is determined from the multiple candidate pixels based on multiple candidate disparities corresponding to the multiple candidate pixels, wherein the difference between the candidate disparity corresponding to the false detection pixel and the target candidate disparity is greater than a predetermined threshold, and the target candidate disparity corresponds to the candidate pixel with the smallest optimal matching cost value with the target matching pixel among the multiple candidate pixels.

12. The method of claim 11, wherein the row of pixels is a first row of pixels, the set of matching pixels is a first set of matching pixels, the on-chip memory includes a first row buffer and a second row buffer, the first row buffer and the second row buffer are configured to operate in a ping-pong buffer mode, wherein the first row buffer is used to store disparity corresponding to the first set of matching pixels, and the second row buffer is used to store disparity corresponding to a second set of matching pixels, the second set of matching pixels being determined based on the second row of pixels in the second image block.

13. The method of claim 1, wherein the first image block and the second image block have overlapping regions, the first image block includes a first non-overlapping region, the second image block includes a second non-overlapping region, and the set of edge pixels includes: A group of pixels adjacent to the overlapping region in the first non-overlapping region, or The overlapping region is a group of pixels adjacent to the second non-overlapping region.

14. The method according to claim 1, wherein a set of edge pixels is a first set of edge pixels, the edge cost information is first edge cost information, and providing the first dynamic matching cost and the second dynamic matching cost includes: The on-chip memory stores second edge cost information corresponding to the second group of edge pixels in the second dynamic matching cost, wherein the second group of edge pixels includes pixels in the second image block that are adjacent to the third image block among the plurality of image blocks; Based on the second edge cost information obtained from the on-chip memory, a third dynamic matching cost corresponding to the predetermined disparity range between the third image block and the second image is determined; as well as The first dynamic matching cost, the second dynamic matching cost, and the third dynamic matching cost are provided to determine the disparity map between the first image and the second image.

15. A processing circuit, the processing circuit including an on-chip memory, characterized in that, The processing circuit is configured to perform the method according to any one of claims 1 to 14.

16. An electronic device, characterized in that, The electronic device includes processing circuitry configured to perform the method according to any one of claims 1 to 14.

17. A computer-readable storage medium storing instructions that, when executed by a computing device, cause the computing device to perform the method according to any one of claims 1 to 14.

18. A computer program product comprising instructions that, when executed by a computing device, cause the computing device to perform the method according to any one of claims 1 to 14.

Images