Stereo matching implementation method and apparatus for real-time depth estimation of mobile devices

Abstract

The application relates to a stereo matching implementation method and device for real-time depth estimation of a mobile device, which comprises the following steps: performing matching cost calculation on a target image pair to obtain a cost matrix corresponding to a main image in the target image pair; based on the cost matrix, adopting a reverse path synchronous aggregation mode to determine the aggregation cost of each pixel in the main image under each disparity; according to the aggregation cost, adopting an expanded thread data slice mode to perform disparity calculation on each pixel in the main image; and performing disparity optimization on each pixel in the main image to obtain a disparity map corresponding to the main image. In the cost fusion calculation, the cost aggregation is simultaneously performed on two paths which are reverse paths of each other, so that the access cost of the global memory is reduced. Meanwhile, in the disparity calculation, the number of concurrent threads is reduced through the expanded thread data slice mode, and then the thread synchronization cost is reduced.

Description

Technical Field

[0001] This invention relates to computer stereo vision, and more particularly to a method and apparatus for stereo matching for real-time depth estimation of mobile devices. Background Technology

[0002] Stereo matching is an important and very difficult problem in computer vision. Its goal is to find matching points in images from different viewpoints. Stereo matching can be considered the foundation of depth estimation.

[0003] Stereo vision matching is typically performed on desktop graphics processing units (GPUs). Researchers attempted stereo vision matching on mobile GPUs, but the performance was even worse than running on a mobile CPU. This is due to the following limitations of GPUs:

[0004] (1) Limited memory bandwidth. The most time-consuming steps in stereo matching are matching cost calculation and cost aggregation, as they require frequent memory read / write operations to calculate the disparity for each pixel. However, mobile graphics processors have very limited memory bandwidth compared to desktop graphics processors. Therefore, the frequent memory read / write operations in stereo matching result in longer latency on mobile graphics processors.

[0005] (2) Limited memory architecture support. In cost aggregation for stereo vision matching, thread synchronization is required to obtain the minimum cost aggregation of a pixel across all disparities. In desktop graphics processors, thread synchronization is performed in fast on-chip shared memory, while mobile graphics processors only have off-chip shared memory, which is much slower.

[0006] In summary, due to the limited memory resources of mobile graphics processors, using mobile graphics processors for stereo matching results in poor real-time parallax calculation performance, which in turn prevents mobile devices from achieving real-time depth estimation. Summary of the Invention

[0007] The purpose of this invention is to provide a method and apparatus for stereo matching in real-time depth estimation of mobile devices, so as to solve the problem of poor real-time disparity calculation performance caused by using mobile graphics processors for stereo matching, and reduce the latency of stereo matching by mobile graphics processors.

[0008] In a first aspect, the present invention provides a stereo matching implementation method for real-time depth estimation of mobile devices, the method comprising:

[0009] The matching cost is calculated for the target image pair to obtain the cost matrix corresponding to the main image in the target image pair; wherein, the cost matrix corresponding to the main image stores the matching cost of each pixel in the main image under each disparity within the disparity range;

[0010] Based on the cost matrix, the cost of each pixel in the main image is aggregated using the reverse path synchronous aggregation method to obtain the aggregated cost of each pixel in the main image under each disparity.

[0011] Based on the aggregate cost of each pixel in the main image under each disparity, the disparity is calculated for each pixel in the main image by expanding the thread data slice.

[0012] Perform disparity optimization on each pixel in the main image to obtain the disparity map corresponding to the main image;

[0013] The target image pair is obtained by simultaneous capture by dual cameras of a mobile device.

[0014] According to the present invention, a stereo matching implementation method for real-time depth estimation of mobile devices includes calculating the matching cost of target image pairs to obtain the cost matrix corresponding to the main image in the target image pair, comprising:

[0015] The Census transform method is used to convert each pixel in the sub-image of the main image and the target image pair into a binary string;

[0016] For each pixel pair encompassed by the main image and the sub-image, the binary string of the corresponding pixel pair is read, and the SGBM algorithm is used to process the read binary string to obtain the Hamming distance between the two pixels in the corresponding pixel pair.

[0017] Based on the Hamming distance between two pixels in all pixel pairs and the pre-stored matching cost smoothing algorithm, the matching cost value of each pixel in the main image under each disparity is calculated and stored.

[0018] Wherein, the pixel pair is composed of pixel Φ in the main image and the corresponding pixel in the sub-image with a disparity distance D between them. i The pixels constitute the D; i It is within the parallax range.

[0019] According to the present invention, a stereo matching implementation method for real-time depth estimation of mobile devices is provided, wherein two paths that are opposite to each other flow in opposite directions but have the same content; the step of performing cost aggregation on each pixel in the main image based on the cost matrix using a synchronous aggregation method of opposite paths to obtain the aggregated cost of each pixel in the main image under each disparity includes:

[0020] Based on the cost matrix, cost aggregation calculation is performed synchronously on two pixels with the same index value in two paths that are opposite to each other, so as to obtain the aggregated cost of each pixel in the main image under each disparity and under each path.

[0021] The aggregate cost of each pixel in the main image under each disparity and under each path is added together to obtain the aggregate cost of each pixel in the main image under each disparity.

[0022] The index value indicates the sorting number of the pixels along the path.

[0023] According to the stereo matching implementation method for real-time depth estimation of mobile devices provided by the present invention, after synchronously performing cost aggregation calculation on two pixels with the same index value in two mutually opposite paths, the method further includes:

[0024] The aggregated costs of two pixels with the same index value in the two paths that are opposite to each other are merged into an array and stored.

[0025] According to the present invention, a stereo matching implementation method for real-time depth estimation of mobile devices is provided, wherein the disparity calculation is performed on each pixel of the main image by expanding the thread data slices based on the aggregate cost of each pixel in the main image under each disparity, including:

[0026] Expand the data slice for each thread to 2. n ;

[0027] For each pixel in the main image, using Each thread reads the aggregate cost of the corresponding pixel under all disparities, so that each thread reads its own 2 n The disparity corresponding to the minimum aggregation cost is obtained by comparing the aggregation costs.

[0028] Will The minimum disparity obtained by each thread is used as the disparity of the corresponding pixel;

[0029] Wherein, D is the total number of parallaxes covered by the parallax range, and n is a preset value.

[0030] According to the present invention, a stereo matching implementation method for real-time depth estimation of mobile devices is provided, wherein the matching cost calculation, the cost aggregation and the disparity calculation are performed on the GPU of the mobile device; and the disparity optimization is performed on the CPU of the mobile device.

[0031] According to the present invention, a stereo matching implementation method for real-time depth estimation of mobile devices is provided, wherein n is set to 3.

[0032] Secondly, the present invention also provides a stereo matching implementation device for real-time depth estimation of mobile devices, the device comprising:

[0033] The matching cost calculation module is used to calculate the matching cost of the target image pair to obtain the cost matrix corresponding to the main image in the target image pair; wherein, the cost matrix corresponding to the main image stores the matching cost of each pixel in the main image under each disparity within the disparity range;

[0034] The cost aggregation module is used to perform cost aggregation on each pixel in the main image based on the cost matrix and using the reverse path synchronous aggregation method to obtain the aggregated cost of each pixel in the main image under each disparity.

[0035] The disparity calculation module is used to calculate the disparity of each pixel in the main image by expanding the thread data slice based on the aggregation cost of each pixel in the main image under each disparity.

[0036] The disparity optimization module is used to perform disparity optimization on each pixel in the main image to obtain the disparity map corresponding to the main image.

[0037] The target image pair is obtained by simultaneous capture by dual cameras of a mobile device.

[0038] Thirdly, the present invention also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the stereo matching method for real-time depth estimation of a mobile device as described in the first aspect.

[0039] Fourthly, the present invention also discloses a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the stereo matching implementation method for real-time depth estimation of mobile devices as described in the first aspect. The present invention provides a stereo matching implementation method and apparatus for real-time depth estimation of mobile devices, which calculates the matching cost of a target image pair to obtain a cost matrix corresponding to the main image in the target image pair; wherein the cost matrix corresponding to the main image stores the matching cost value of each pixel in the main image under each disparity within the disparity range; based on the cost matrix, a reverse path synchronous aggregation method is used to aggregate the cost of each pixel in the main image to obtain the aggregated cost of each pixel in the main image under each disparity; according to the aggregated cost of each pixel in the main image under each disparity, a method of expanding thread data slicing is used to calculate the disparity of each pixel in the main image; and disparity optimization is performed on each pixel in the main image to obtain a disparity map corresponding to the main image. In the cost fusion calculation step of stereo matching, this invention simultaneously performs cost aggregation on two mutually opposite paths (e.g., a left-to-right aggregation path and a right-to-left aggregation path, or a top-to-bottom aggregation path and a bottom-to-top aggregation path) to reduce the overhead of accessing global memory. Simultaneously, in the disparity calculation step of stereo matching, the number of concurrent threads is reduced by expanding the data slices on the thread, thereby reducing the overhead of thread synchronization. When performing image processing on mobile devices, the reduction in global memory access overhead and thread synchronization overhead lowers the latency of stereo matching, thus improving the real-time performance of depth estimation. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0041] Figure 1 This is a flowchart of a stereo matching implementation method for real-time depth estimation of mobile devices provided by the present invention;

[0042] Figure 2 This is a schematic diagram illustrating the process of reverse path synchronous aggregation and aggregation cost merging and storage provided by the present invention;

[0043] Figure 3 This is a schematic diagram of data slicing provided by the present invention;

[0044] Figure 4This is a schematic diagram of a stereo matching implementation device for real-time depth estimation of mobile devices provided by the present invention;

[0045] Figure 5 This is a schematic diagram of an electronic device structure for implementing a stereo matching method for real-time depth estimation of mobile devices, as provided by the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0047] The following is combined Figures 1 to 5 This invention describes a stereo matching implementation method and apparatus for real-time depth estimation of mobile devices.

[0048] In a first aspect, the present invention provides a stereo matching implementation method for real-time depth estimation of mobile devices, such as... Figure 1 As shown, the method includes:

[0049] S11. Perform matching cost calculation on the target image pair to obtain the cost matrix corresponding to the main image in the target image pair; wherein, the cost matrix corresponding to the main image stores the matching cost of each pixel in the main image under each disparity within the disparity range;

[0050] S12. Based on the cost matrix, the cost of each pixel in the main image is aggregated using the reverse path synchronous aggregation method to obtain the aggregated cost of each pixel in the main image under each disparity.

[0051] S13. Based on the aggregation cost of each pixel in the main image under each disparity, the disparity is calculated for each pixel in the main image by expanding the thread data slice.

[0052] S14. Perform disparity optimization on each pixel in the main image to obtain the disparity map corresponding to the main image;

[0053] The target image pair is obtained by simultaneous capture by dual cameras of a mobile device.

[0054] Current state-of-the-art stereo matching algorithms and CNN models, such as Semi-Global-Block Matching (SGBM), StereoNet, and HITNet, are computationally intensive and have long runtimes when implemented on mobile devices. Furthermore, stereo matching requires online processing to find correspondences between points in a frame. This long runtime leads to low refresh rates in depth estimation applications. Therefore, the goal of this invention is to accelerate stereo matching on computationally limited mobile devices.

[0055] Given the high accuracy and efficiency of SGBM, it is primarily used for stereo matching. However, experiments revealed that the speed at which SGBM generates disparity maps for stereo matching on mobile devices is insufficient to meet the demands of real-time depth estimation. Researchers concluded that the limited memory bandwidth of mobile GPUs reduces computational efficiency. Therefore, this invention employs reverse path synchronization in SGBM to perform cost aggregation, reducing the overhead of accessing global memory; simultaneously, it expands thread data slices to reduce the number of concurrent threads. Through these methods, the present invention significantly reduces the memory access overhead and synchronization costs between threads on mobile GPUs, decreases the use of mobile GPU memory resources, and improves the computational efficiency of stereo matching on mobile GPUs.

[0056] This invention provides a stereo matching implementation method for real-time depth estimation on mobile devices. In the cost fusion calculation step of stereo matching, cost aggregation is performed simultaneously on two paths that are opposite to each other (e.g., a left-to-right aggregation path and a right-to-left aggregation path, or a top-to-bottom aggregation path and a bottom-to-top aggregation path) to reduce the overhead of accessing global memory. Simultaneously, in the disparity calculation step of stereo matching, the number of concurrent threads is reduced by expanding the data slices on the thread, thereby reducing the overhead of thread synchronization. The reduction in global memory access overhead and thread synchronization overhead lowers the latency of stereo matching, thus improving the real-time performance of depth estimation.

[0057] Based on the above embodiments, as an optional embodiment, the step of calculating the matching cost for the target image pair to obtain the cost matrix corresponding to the main image in the target image pair includes:

[0058] The Census transform method is used to convert each pixel in the sub-image of the main image and the target image pair into a binary string;

[0059] For each pixel pair encompassed by the main image and the sub-image, the binary string of the corresponding pixel pair is read, and the SGBM algorithm is used to process the read binary string to obtain the Hamming distance between the two pixels in the corresponding pixel pair.

[0060] Based on the Hamming distance between two pixels in all pixel pairs and the pre-stored matching cost smoothing algorithm, the matching cost value of each pixel in the main image under each disparity is calculated and stored.

[0061] Wherein, the pixel pair is composed of pixel Φ in the main image and the corresponding pixel in the sub-image with a disparity distance D between them. i The pixels constitute the D; i It is within the parallax range.

[0062] This embodiment uses the cost matching calculation of the SGBM algorithm as an example for illustration:

[0063] In the SGBM algorithm, each step utilizes the results of the previous step, requiring extensive read and write operations to global memory. In the cost matching calculation step, the Census transform is first used to convert each pixel in the target stereo image pair into a binary string. Then, the Hamming distance between each pixel in the main image and the corresponding pixel within the disparity range in the sub-image is calculated, and the Hamming distance calculation results are written to memory. Finally, a sliding window is used to read the Hamming distances for matching cost smoothing, resulting in the final cost matrix.

[0064] Specifically, the image resolution is set to W×H, and the maximum disparity range is D. Calculating the Hamming distance between a pixel in the main image and a corresponding pixel within the disparity range of that pixel in the sub-image requires reading the binary strings of these two pixels from memory and writing their Hamming distance values ​​back into memory. Therefore, for all pixels in the main image, this requires W×H×D×3 memory read / write operations. Then, matching cost smoothing is performed by reading the Hamming distance through a sliding window. This requires reading the Hamming distance calculation results of two pixels from memory and inputting them into the matching cost smoothing algorithm to obtain the corresponding matching cost. Therefore, obtaining the disparity cost for all pixels in the main image requires a total of W×H×D×(3+3) memory read / write operations.

[0065] To reduce memory read / write overhead, this invention employs a computational fusion approach for cost matching calculation. Specifically, the Hamming distance is not stored after calculation; it is directly used for calculating the matching cost. In this case, the number of memory read / write operations is W×H×D×(4+1), which is a reduction of W×H×D compared to the original method.

[0066] It should be noted that the matching cost smoothing algorithm uses an existing method, which will not be elaborated here.

[0067] Based on the above embodiments, as an optional embodiment, two paths that are reverse paths flow in opposite directions but have the same content; the step of using the reverse path synchronous aggregation method to perform cost aggregation on each pixel in the main image based on the cost matrix to obtain the aggregated cost of each pixel in the main image under each disparity includes:

[0068] Based on the cost matrix, cost aggregation calculation is performed synchronously on two pixels with the same index value in two paths that are opposite to each other, so as to obtain the aggregated cost of each pixel in the main image under each disparity and under each path.

[0069] The aggregate cost of each pixel in the main image under each disparity and under each path is added together to obtain the aggregate cost of each pixel in the main image under each disparity.

[0070] The index value indicates the sorting number of the pixels along the path.

[0071] This embodiment uses the cost aggregation of the SGBM algorithm as an example for illustration:

[0072] Assuming the SGBM algorithm uses four paths for aggregation (aggregation path from left to right, aggregation path from right to left, aggregation path from top to bottom, and aggregation path from bottom to top), during cost aggregation, each pixel needs to calculate the aggregation cost for each aggregation path separately, which will require a large number of memory read and write operations (2×H×D+2×W×D).

[0073] The present invention takes into account that the data sequences read by two paths that are opposite to each other (the aggregation path from left to right and the aggregation path from right to left are opposite to each other, and the aggregation path from top to bottom and the aggregation path from bottom to top are opposite to each other) are consistent, so cost aggregation can be performed on two paths that are opposite to each other at the same time. Figure 2 This is a schematic diagram illustrating the process of synchronous aggregation and merging storage of aggregation costs along the reverse path, as shown below. Figure 2 As shown, pixels p with the same index value in the aggregation path from left to right and the aggregation path from right to left. r and p l Simultaneous calculations are possible. This reduces memory accesses by W*D+H*D; the top-down aggregation path is equivalent to the bottom-up aggregation path. Finally, synchronous aggregation via the reverse path reduces memory accesses by (H×D+W×D).

[0074] Based on the above embodiments, as an optional embodiment, after simultaneously performing cost aggregation calculation on two pixels with the same index value in two mutually opposite paths, the method further includes:

[0075] The aggregated costs of two pixels with the same index value in the two paths that are opposite to each other are merged into an array and stored.

[0076] In this embodiment, data is merged and written into memory to further reduce memory access overhead.

[0077] Specifically, pixels p with the same index value in the left-to-right aggregation path and the right-to-left aggregation path. r and p l The corresponding aggregation cost is merged into an array for memory write operations (e.g., merging two 16-bit values ​​into a single 32-bit value), and it is written only once. The top-down aggregation path is equivalent to the bottom-up aggregation path. Ultimately, this can reduce memory write operations by half.

[0078] In summary, this invention reduces the overhead of memory read / write operations in stereo matching by using computational fusion and data merging to write data into memory.

[0079] Based on the above embodiments, as an optional embodiment, the step of calculating the disparity of each pixel in the main image by expanding the thread data slices according to the aggregation cost of each pixel in the main image under each disparity includes:

[0080] Expand the data slice for each thread to 2. n ;

[0081] For each pixel in the main image, using Each thread reads the aggregate cost of the corresponding pixel under all disparities, so that each thread reads its own 2 n The disparity corresponding to the minimum aggregation cost is obtained by comparing the aggregation costs.

[0082] Will The minimum disparity obtained by each thread is used as the disparity of the corresponding pixel;

[0083] Wherein, D is the total number of parallaxes covered by the parallax range, and n is a preset value.

[0084] This embodiment uses the disparity calculation of the SGBM algorithm as an example for illustration:

[0085] In disparity computation, calculating the minimum aggregation cost for each pixel is a crucial step, which requires synchronizing the aggregation costs across all disparities for each pixel. The SGBM algorithm typically creates a number of threads equal to the maximum number of disparities (D) to compute the minimum aggregation cost for a pixel. However, this leads to significant synchronization overhead and memory access contention. In desktop GPUs, synchronization is performed in on-chip shared memory, resulting in a shorter execution time, while in mobile GPUs, synchronization is performed in off-chip shared memory, which is more time-consuming.

[0086] This invention addresses the above problems by using a method of expanding data slices. Figure 3 This is a schematic diagram of data slicing provided by the present invention, such as... Figure 3 As shown, the present invention uses vectorized memory access to obtain 2 [units / items] at a time. n We take 10 consecutive data points and then perform single-threaded vectorization processing on them. By using an extended data slicing method, each thread computes 2... n The aggregation cost is calculated, and the minimum aggregation cost on that thread is obtained. Therefore, synchronization only needs to be performed in the moving shared memory. By using threads, the minimum value of all differences can be obtained, which greatly reduces the data synchronization cost on mobile GPUs. Furthermore, this method can reduce the number of shared memory reads and writes. Through vectorized memory access, this embodiment only uses 2... n The disparity value is read into memory once, which reduces the number of memory reads. And only need to Writing data to shared memory minimizes aggregation costs. This invention increases the amount of data per thread, improving data throughput and bandwidth utilization, and allows cost calculation for a new frame to begin immediately after the disparity calculation for the last frame is completed.

[0087] Based on the above embodiments, as an optional embodiment, the matching cost calculation, the cost aggregation, and the disparity calculation are performed on the GPU of the mobile device; and the disparity optimization is performed on the CPU of the mobile device.

[0088] Considering that disparity optimization is a relatively independent step, to further reduce the latency of stereo matching on mobile devices, we allocate frame disparity optimization to the CPU. The other three steps—cost matching calculation, cost aggregation, and disparity calculation—still run on the mobile GPU. That is, once the disparity calculation for each frame pair is completed on the GPU, the calculation for the next frame pair can begin.

[0089] Based on the above embodiments, as an optional embodiment, n is set to 3.

[0090] In this invention, n cannot exceed a set threshold. If the set threshold is exceeded, the data processing speed of a single thread will be greatly reduced, which is not conducive to parallax calculation.

[0091] It should be noted that the value of n may vary for different mobile devices, so the optimal value of n should be found first in practical applications.

[0092] In the stereo matching verification stage of mobile devices, the SGBM of this invention was used for stereo matching on both HWM40P and HWP30, while D-SGBM (an SGBM developed for desktop GPUs) was used on both HWM40P and HWP30. Table 1 shows a latency comparison between the SGBM of this invention and D-SGBM. In Table 1, "—" indicates that the latency is hidden by processing simultaneously through two threads of CPU and GPU. As can be seen from Table 1, the SGBM of this invention reduces latency by 81.4% and 72.8% on HWM40P and HWP30, respectively, compared to D-SGBM. Furthermore, each step has been significantly optimized.

[0093] Table 1

[0094]

[0095] Secondly, the stereo matching implementation apparatus for real-time depth estimation of mobile devices provided by the present invention will be described. The stereo matching implementation apparatus for real-time depth estimation of mobile devices described below and the stereo matching implementation method for real-time depth estimation of mobile devices described above can be referred to in correspondence with each other. Figure 4 This example illustrates a structural diagram of a stereo matching implementation device for real-time depth estimation on mobile devices. The device includes:

[0096] The matching cost calculation module 21 is used to calculate the matching cost of the target image pair to obtain the cost matrix corresponding to the main image in the target image pair; wherein, the cost matrix corresponding to the main image stores the matching cost of each pixel in the main image under each disparity within the disparity range.

[0097] The cost aggregation module 22 is used to perform cost aggregation on each pixel in the main image based on the cost matrix and using the reverse path synchronous aggregation method to obtain the aggregated cost of each pixel in the main image under each disparity.

[0098] The disparity calculation module 23 is used to calculate the disparity of each pixel in the main image by expanding the thread data slice based on the aggregation cost of each pixel in the main image under each disparity.

[0099] The disparity optimization module 24 is used to perform disparity optimization on each pixel in the main image to obtain a disparity map corresponding to the main image.

[0100] The target image pair is obtained by simultaneous capture by dual cameras of a mobile device.

[0101] This invention provides a stereo matching implementation device for real-time depth estimation on mobile devices. In the cost fusion calculation step of stereo matching, cost aggregation is simultaneously performed on two paths that are opposite to each other (e.g., a left-to-right aggregation path and a right-to-left aggregation path, or a top-to-bottom aggregation path and a bottom-to-top aggregation path) to reduce the overhead of accessing global memory. Simultaneously, in the disparity calculation step of stereo matching, the number of concurrent threads is reduced by expanding the data slices on the threads, thereby reducing the overhead of thread synchronization. The reduction in global memory access overhead and thread synchronization overhead lowers the latency of stereo matching, thus improving the real-time performance of depth estimation.

[0102] The stereo matching implementation apparatus for real-time depth estimation of mobile devices provided in this embodiment of the invention specifically executes the flow of the stereo matching implementation method embodiments for real-time depth estimation of mobile devices described above. For details, please refer to the content of the stereo matching implementation method embodiments for real-time depth estimation of mobile devices described above, which will not be repeated here.

[0103] Thirdly, Figure 5 A schematic diagram of the physical structure of an electronic device is shown. (For example...) Figure 5As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call logic instructions in the memory 530 to execute a stereo matching implementation method for real-time depth estimation of a mobile device. This method includes: calculating the matching cost of a target image pair to obtain a cost matrix corresponding to the main image in the target image pair; wherein the cost matrix corresponding to the main image stores the matching cost value of each pixel in the main image under each disparity within the disparity range; based on the cost matrix, performing cost aggregation on each pixel in the main image using a reverse path synchronous aggregation method to obtain the aggregated cost of each pixel in the main image under each disparity; calculating the disparity of each pixel in the main image using an expanded thread data slice method based on the aggregated cost of each pixel in the main image under each disparity; and performing disparity optimization on each pixel in the main image to obtain a disparity map corresponding to the main image; wherein the target image pair is obtained by simultaneous capture by dual cameras of the mobile device.

[0104] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0105] Fourthly, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, it performs a stereo matching implementation method for real-time depth estimation of a mobile device, the method comprising: calculating a matching cost for a pair of target images to obtain a cost matrix corresponding to a main image in the pair of target images; wherein the cost matrix corresponding to the main image stores the matching cost value of each pixel in the main image under each disparity within the disparity range; based on the cost matrix, performing cost aggregation on each pixel in the main image using a reverse path synchronous aggregation method to obtain an aggregated cost of each pixel in the main image under each disparity; calculating the disparity of each pixel in the main image using an expanded thread data slice method according to the aggregated cost of each pixel in the main image under each disparity; performing disparity optimization on each pixel in the main image to obtain a disparity map corresponding to the main image; wherein the pair of target images is obtained by simultaneous capture by dual cameras of a mobile device.

[0106] Fifthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon for executing a stereo matching implementation method for real-time depth estimation of a mobile device. The method includes: calculating a matching cost for a pair of target images to obtain a cost matrix corresponding to a main image in the pair of target images; wherein the cost matrix corresponding to the main image stores the matching cost value of each pixel in the main image under each disparity within a disparity range; based on the cost matrix, performing cost aggregation on each pixel in the main image using a reverse path synchronous aggregation method to obtain an aggregated cost for each pixel in the main image under each disparity; calculating the disparity for each pixel in the main image using an expanded thread data slice method based on the aggregated cost of each pixel in the main image under each disparity; and optimizing the disparity for each pixel in the main image to obtain a disparity map corresponding to the main image; wherein the pair of target images is obtained by simultaneous capture by dual cameras of a mobile device.

[0107] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0108] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for stereo matching implementation for real-time depth estimation of mobile devices, characterized in that, The method includes: The matching cost is calculated for the target image pair to obtain the cost matrix corresponding to the main image in the target image pair; wherein, the cost matrix corresponding to the main image stores the matching cost of each pixel in the main image under each disparity within the disparity range; Based on the cost matrix, the cost of each pixel in the main image is aggregated using the reverse path synchronous aggregation method to obtain the aggregated cost of each pixel in the main image under each disparity. Based on the aggregate cost of each pixel in the main image under each disparity, the disparity is calculated for each pixel in the main image by expanding the thread data slice. Perform disparity optimization on each pixel in the main image to obtain the disparity map corresponding to the main image; The target image pair is obtained by simultaneous capture by dual cameras of a mobile device; Two paths that are opposite to each other flow in opposite directions and read the same data sequence; Based on the cost matrix, a reverse path synchronous aggregation method is used to aggregate the cost of each pixel in the main image to obtain the aggregated cost of each pixel in the main image under each disparity, including: Based on the cost matrix, cost aggregation calculation is performed synchronously on two pixels with the same index value in two paths that are opposite to each other, so as to obtain the aggregated cost of each pixel in the main image under each disparity and under each path. The aggregate cost of each pixel in the main image under each disparity and under each path is added together to obtain the aggregate cost of each pixel in the main image under each disparity. The index value indicates the sorting number of the pixels along the path direction; After simultaneously performing cost aggregation calculations on two pixels with the same index value in two mutually opposite paths, the process further includes: The aggregated costs of two pixels with the same index value in the two paths that are opposite to each other are merged into an array and stored.

2. The method for stereo matching implementation for real-time depth estimation of mobile devices according to claim 1, wherein, The step of calculating the matching cost for the target image pair to obtain the cost matrix corresponding to the main image in the target image pair includes: The Census transform method is used to convert each pixel in the sub-image of the main image and the target image pair into a binary string; For each pixel pair encompassed by the main image and the sub-image, the binary string of the corresponding pixel pair is read, and the SGBM algorithm is used to process the read binary string to obtain the Hamming distance between the two pixels in the corresponding pixel pair. Based on the Hamming distance between two pixels in all pixel pairs and the pre-stored matching cost smoothing algorithm, the matching cost value of each pixel in the main image under each disparity is calculated and stored. Wherein, the pixel pair is composed of a pixel in the main image and a pixel in the sub-image which is apart from the corresponding pixel of the pixel by a disparity ; the disparity is within a disparity range.

3. The method for stereo matching implementation for real-time depth estimation of mobile devices according to claim 1, wherein, The step of calculating disparity for each pixel in the main image based on the aggregate cost of each pixel under each disparity, using an expanded thread data slice approach, includes: Expand the data slice of each thread to ; For each pixel in the main image, using Each thread reads the aggregated cost of the corresponding pixel across all disparities, so that each thread reads its own cost. The disparity corresponding to the minimum aggregation cost is obtained by comparing the aggregation costs. The minimum disparity obtained by the one thread is taken as the disparity of the corresponding pixel. ​ Wherein, the is the total number of parallax values covered by the parallax range, is a preset value.

4. The method for stereo matching implementation for real-time depth estimation of mobile devices of claim 1, wherein, The matching cost calculation, the cost aggregation, and the disparity calculation are performed on the GPU of the mobile device; the disparity optimization is performed on the CPU of the mobile device.

5. The method for stereo matching implementation for real-time depth estimation of mobile devices according to claim 3, wherein, The is set to 3.

6. A device for stereo matching implementation for real-time depth estimation of mobile devices, characterized in that, The device includes: The matching cost calculation module is used to calculate the matching cost of the target image pair to obtain the cost matrix corresponding to the main image in the target image pair; wherein, the cost matrix corresponding to the main image stores the matching cost of each pixel in the main image under each disparity within the disparity range; The cost aggregation module is used to perform cost aggregation on each pixel in the main image based on the cost matrix and using the reverse path synchronous aggregation method to obtain the aggregated cost of each pixel in the main image under each disparity. The disparity calculation module is used to calculate the disparity of each pixel in the main image by expanding the thread data slice based on the aggregation cost of each pixel in the main image under each disparity. The disparity optimization module is used to perform disparity optimization on each pixel in the main image to obtain the disparity map corresponding to the main image. The target image pair is obtained by simultaneous capture by dual cameras of a mobile device; Two paths that are opposite to each other flow in opposite directions and read the same data sequence; Based on the cost matrix, a reverse path synchronous aggregation method is used to aggregate the cost of each pixel in the main image to obtain the aggregated cost of each pixel in the main image under each disparity, including: Based on the cost matrix, cost aggregation calculation is performed synchronously on two pixels with the same index value in two paths that are opposite to each other, so as to obtain the aggregated cost of each pixel in the main image under each disparity and under each path. The aggregate cost of each pixel in the main image under each disparity and under each path is added together to obtain the aggregate cost of each pixel in the main image under each disparity. The index value indicates the sorting number of the pixels along the path direction; After simultaneously performing cost aggregation calculations on two pixels with the same index value in two mutually opposite paths, the process further includes: The aggregated costs of two pixels with the same index value in the two paths that are opposite to each other are merged into an array and stored.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements a stereo matching method for real-time depth estimation of mobile devices as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by the processor, it implements a stereo matching method for real-time depth estimation of mobile devices as described in any one of claims 1 to 5.

Images