Stereo matching method and system based on binocular camera

Abstract

The present invention discloses a stereo matching method and system based on a binocular camera, the method comprising: collecting multiple consecutive frames of left and right eye images in a target area; performing disparity matching cost calculation on each of the left and right eye images to obtain multiple disparity matching cost values; performing multi-directional aggregation on each of the disparity matching cost values, and calculating by argmax (maximum independent variable point set) to obtain a disparity map corresponding to the maximum disparity matching cost value; normalizing the disparity map, and performing sub-pixel disparity calculation to obtain a stereo matching result. Effective deployment in a deep learning chip is achieved, and time and effect are met, with better generalization ability, and better effect in sub-pixel accuracy, which improves the algorithm accuracy and generalization ability of stereo matching, thereby providing an algorithm basis for improving the accuracy of autonomous driving data.

Description

Technical Field

[0001] The present invention relates to the technical field of assisted driving, and in particular to a stereo matching method and system based on a binocular camera. Background Art

[0002] As people's demand for safer and more convenient travel grows, intelligent driving technology is booming. The ability to perceive and understand the environment is the basis and premise of the automotive intelligent system. Intelligent vehicles collect views through binocular cameras, analyze the surrounding environment after perceiving it, and detect driving conditions by providing information to the control system.

[0003] When performing depth estimation on the stereo images captured by the binocular camera, stereo matching is required. Its purpose is to estimate the correspondence between all pixels between the two corrected images. The performance of stereo matching directly affects the accuracy of autonomous driving information recognition. Summary of the invention

[0004] To this end, an embodiment of the present invention provides a stereo matching method and system based on a binocular camera to improve the algorithm accuracy and generalization ability of stereo matching, thereby providing an algorithmic basis for improving the accuracy of autonomous driving data.

[0005] In order to achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] A stereo matching method based on a binocular camera, the method comprising:

[0007] Collect multiple consecutive frames of left and right eye images in the target area;

[0008] Performing disparity matching cost calculation on each of the left and right eye images to obtain a plurality of disparity matching cost values;

[0009] Aggregating the disparity matching cost values ​​in multiple directions, and calculating by argmax (maximum independent variable point set), so as to obtain a disparity map corresponding to the maximum disparity matching cost value;

[0010] The disparity map is normalized and sub-pixel disparity is calculated to obtain a stereo matching result.

[0011] Furthermore, a plurality of consecutive frames of left and right eye images in the target area are collected, and then the method further includes:

[0012] The convolution operator of the deep learning network is used to perform Gaussian filtering calculation on the left and right eye images to extract high-dimensional feature information of the images and improve the accuracy of matching cost calculation.

[0013] Furthermore, the following formula is used to calculate the disparity matching cost of each of the left and right images:

[0014]

[0015] Among them, x is the grayscale value of the pixel in the left image, and y is the grayscale value of the pixel in the right image. is the mean value of x, is the y-mean, is the variance of x, is the variance of y, is the covariance of x and y, and is a constant.

[0016] Furthermore, the constant is calculated using the following formula and :

[0017]

[0018] Where L is the dynamic range of pixel values, , .

[0019] Furthermore, the following formula is used to aggregate the disparity matching cost values ​​in multiple directions:

[0020] in, represents the aggregate cost value in the propagation direction of r, where r represents the propagation direction. represents the matching cost, represents the aggregation cost of p pixel point on d disparity in the r propagation direction, represents the aggregation cost of the previous pixel of pixel p in the propagation direction r on the disparity d, represents the aggregation cost of the previous pixel of pixel p in the propagation direction r on the d-1 disparity, represents the aggregation cost of the previous pixel of pixel p in the propagation direction r at the disparity of d+1. represents the aggregation cost of the previous pixel of pixel p in the propagation direction r with the minimum aggregation cost among all disparities, and P1 and P2 represent penalty terms.

[0021] Furthermore, the disparity map is normalized and sub-pixel disparity is calculated to obtain a stereo matching result, specifically including:

[0022] Based on the disparity map, a softmax normalization process is performed on each pixel to obtain a normalized result corresponding to the position;

[0023] Multiply the disparity value of each point by the normalized result of the corresponding position to obtain the sub-pixel disparity value;

[0024] The sub-pixel disparity value is used as a stereo matching result.

[0025] The present invention also provides a stereo matching system based on a binocular camera, the system comprising:

[0026] An image acquisition unit, used for acquiring multiple consecutive frames of left and right eye images in the target area;

[0027] A cost value calculation unit, used for performing disparity matching cost calculation on each of the left and right eye images to obtain a plurality of disparity matching cost values;

[0028] A cost value aggregation unit, configured to aggregate the disparity matching cost values ​​in multiple directions, and calculate the disparity map corresponding to the maximum disparity matching cost value through argmax (maximum independent variable point set);

[0029] The result output unit is used to normalize the disparity map and perform sub-pixel disparity calculation to obtain a stereo matching result.

[0030] Furthermore, the image acquisition unit is specifically used for:

[0031] Use the convolution operator of the deep learning network to perform Gaussian filtering calculation on the left and right eye images. Use the convolution operator of the deep learning network to perform Gaussian filtering calculation on the left and right eye images to extract high-dimensional feature information of the images and improve the accuracy of matching cost calculation.

[0032] The present invention also provides an intelligent terminal, which includes: a data acquisition device, a processor and a memory;

[0033] The data acquisition device is used to acquire data; the memory is used to store one or more program instructions; and the processor is used to execute one or more program instructions to execute the method as described above.

[0034] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium contains one or more program instructions, and the one or more program instructions are used to execute the method as described above.

[0035] The stereo matching method based on binocular camera provided by the present invention collects multiple consecutive frames of left and right eye images in the target area, calculates the disparity matching cost of each of the left and right eye images to obtain multiple disparity matching cost values; aggregates each of the disparity matching cost values ​​in multiple directions, and calculates by argmax to obtain a disparity map corresponding to the maximum disparity matching cost value; normalizes the disparity map, and performs sub-pixel disparity calculation to obtain a stereo matching result. In this way, the stereo matching method based on binocular camera provided by the present invention realizes effective deployment in deep learning chips through matching cost calculation, matching cost aggregation, disparity calculation and disparity refinement, and meets time and effect, has better generalization ability, and has better effect in sub-pixel accuracy, improves the algorithm accuracy and generalization ability of stereo matching, and thus provides an algorithmic basis for improving the accuracy of autonomous driving data. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0037] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0038] Figure 1 A flowchart of a specific implementation of the stereo matching method based on a binocular camera provided by the present invention;

[0039] Figure 2 The present invention provides a block diagram of a stereo matching system based on a binocular camera according to a specific embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] In order to adapt to deep learning chips and maintain the generalization ability and accuracy of the algorithm, the stereo matching method based on binocular cameras provided by the present invention includes matching cost calculation, matching cost aggregation, disparity calculation and disparity refinement. It uses convolution to perform SSIM feature calculation on the image and calculates the matching cost value, then uses convolution and maximum pooling to aggregate the matching costs, and uses softargmax to obtain the final sub-pixel disparity, which ensures the accuracy and generalization ability of the algorithm, and can be effectively deployed in deep learning chips to provide an algorithmic basis for improving the accuracy of autonomous driving data.

[0042] In a specific embodiment, Figure 1 As shown, the stereo matching method based on binocular cameras provided by the present invention comprises the following steps:

[0043] S1: Collect multiple consecutive frames of left and right eye images in the target area. In order to improve the image effect, use the convolution operator of the deep learning network to perform Gaussian filtering calculation on the left and right eye images to extract high-dimensional feature information of the image and improve the accuracy of matching cost calculation.

[0044] S2: Calculate the disparity matching cost for each of the left and right images to obtain a plurality of disparity matching cost values.

[0045] Specifically, in order to ensure the calculation accuracy of the disparity matching cost value, it is preferred to use the following formula to calculate the disparity matching cost for each of the left and right images:

[0046]

[0047] Among them, x is the grayscale value of the pixel in the left image, and y is the grayscale value of the pixel in the right image. is the mean value of x, is the y-mean, is the variance of x, is the variance of y, is the covariance of x and y, and is a constant.

[0048] Calculate the constant using the following formula and :

[0049]

[0050] Where L is the dynamic range of pixel values, , .

[0051] Specifically, the range of structural similarity is -1 to 1. As an implementation of the structural similarity theory, the structural similarity index defines structural information as independent of brightness and contrast from the perspective of image composition, reflecting the properties of the structure of objects in the scene, and models the distortion as a combination of three different factors: brightness, contrast, and structure.

[0052] S3: Aggregating the disparity matching cost values ​​in multiple directions and calculating by argmax (maximum independent variable point set) to obtain a disparity map corresponding to the maximum disparity matching cost value.

[0053] In actual usage scenarios, the following formula can be used to aggregate the disparity matching cost values ​​in multiple directions:

[0054]

[0055] in, represents the aggregate cost value in the propagation direction of r, where r represents the propagation direction. represents the matching cost, represents the aggregation cost of p pixel point on d disparity in the r propagation direction, represents the aggregation cost of the previous pixel of pixel p in the propagation direction r on the disparity d, represents the aggregation cost of the previous pixel of pixel p in the propagation direction r on the d-1 disparity, represents the aggregation cost of the previous pixel of pixel p in the propagation direction r at the disparity of d+1. represents the aggregation cost of the previous pixel of pixel p in the propagation direction r with the minimum aggregation cost among all disparities, and P1 and P2 represent penalty terms.

[0056] Specifically, the costs of different directions can be aggregated to obtain the final aggregated cost value. In the present invention, the convolution operation is used to complete the same propagation process. Taking the process from left to right as an example, first, in the previous step, , divide it into H pieces of size , using a matrix of size equal to the maximum disparity for each The convolution kernel is convolved, where the penalty items of P1 and P2 are converted from addition to multiplication. The weight of the convolution kernel is composed of the penalty items. For different disparities d, the corresponding penalty item weights are set to form the convolution kernel, and finally a The result after the spread of.

[0057] S4: normalizing the disparity map and performing sub-pixel disparity calculation to obtain a stereo matching result.

[0058] Step S4 specifically includes:

[0059] Based on the disparity map, a softmax normalization process is performed on each pixel to obtain a normalized result corresponding to the position;

[0060] Multiply the disparity value of each point by the normalized result of the corresponding position to obtain the sub-pixel disparity value;

[0061] The sub-pixel disparity value is used as a stereo matching result.

[0062] In actual usage scenarios, the purpose of stereo matching is to estimate the correspondence between all pixels in two rectified images. Given a pair of rectified stereo images, the purpose of disparity estimation is to calculate the disparity d of each pixel in the reference image. Disparity refers to the horizontal displacement between a pair of corresponding points in the reference image and the target image. For a pixel (x, y) in the reference image, if the corresponding pixel is found in the target image (xd, y), then the depth of this point can be calculated by f*b / d, where f is the focal length of the camera and b is the distance between the two cameras.

[0063] In the above specific implementation, the stereo matching method based on binocular camera provided by the present invention collects multiple consecutive frames of left and right eye images in the target area, calculates the disparity matching cost for each of the left and right eye images to obtain multiple disparity matching cost values; aggregates each of the disparity matching cost values ​​in multiple directions, and calculates by argmax to obtain the disparity map corresponding to the maximum disparity matching cost value; normalizes the disparity map, and performs sub-pixel disparity calculation to obtain a stereo matching result. In this way, the stereo matching method based on binocular camera provided by the present invention is effectively deployed in a deep learning chip through matching cost calculation, matching cost aggregation, disparity calculation and disparity refinement, and meets the time and effect, has better generalization ability, and has better effect in sub-pixel accuracy, improves the algorithm accuracy and generalization ability of stereo matching, and thus provides an algorithmic basis for improving the accuracy of autonomous driving data.

[0064] In addition to the above method, the present invention also provides a stereo matching system based on a binocular camera, such as Figure 2 As shown, the system comprises:

[0065] The image acquisition unit 100 is used to acquire multiple consecutive frames of left and right eye images in the target area;

[0066] The cost value calculation unit 200 is used to calculate the disparity matching cost of each of the left and right images to obtain a plurality of disparity matching cost values;

[0067] The cost value aggregation unit 300 is used to aggregate the disparity matching cost values ​​in multiple directions and calculate the disparity map corresponding to the maximum disparity matching cost value through argmax (maximum independent variable point set);

[0068] The result output unit 400 is used to normalize the disparity map and perform sub-pixel disparity calculation to obtain a stereo matching result.

[0069] The image acquisition unit is specifically used for:

[0070] The convolution operator of the deep learning network is used to perform Gaussian filtering calculation on the left and right eye images to extract high-dimensional feature information of the images and improve the accuracy of matching cost calculation.

[0071] In the above specific implementation, the stereo matching system based on binocular camera provided by the present invention collects multiple consecutive frames of left and right eye images in the target area, calculates the disparity matching cost for each of the left and right eye images to obtain multiple disparity matching cost values; aggregates each of the disparity matching cost values ​​in multiple directions, and calculates by argmax to obtain the disparity map corresponding to the maximum disparity matching cost value; normalizes the disparity map, and performs sub-pixel disparity calculation to obtain a stereo matching result. In this way, the stereo matching method based on binocular camera provided by the present invention is effectively deployed in a deep learning chip through matching cost calculation, matching cost aggregation, disparity calculation and disparity refinement, and meets the time and effect, has better generalization ability, and has better effect in sub-pixel accuracy, improves the algorithm accuracy and generalization ability of stereo matching, and thus provides an algorithm basis for improving the accuracy of autonomous driving data.

[0072] The present invention also provides an intelligent terminal, which includes: a data acquisition device, a processor and a memory;

[0073] The data acquisition device is used to acquire data; the memory is used to store one or more program instructions; and the processor is used to execute one or more program instructions to execute the method as described above.

[0074] Corresponding to the above embodiment, an embodiment of the present invention further provides a computer-readable storage medium, which contains one or more program instructions. The one or more program instructions are used to be executed by a binocular camera depth calibration system as described above.

[0075] In the embodiment of the present invention, the processor may be an integrated circuit chip with signal processing capability. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0076] The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The processor reads the information in the storage medium and completes the steps of the above method in combination with its hardware.

[0077] The storage medium may be a memory, which may be, for example, a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memory.

[0078] Among them, the non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory.

[0079] The volatile memory may be a random access memory (RAM) which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synclink DRAM (SLDRAM), and direct RAM bus random access memory (DRRAM).

[0080] The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0081] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the present invention can be implemented using a combination of hardware and software. When software is used, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0082] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. A stereo matching method based on a binocular camera, characterized in that: The method comprises: Collect multiple consecutive frames of left and right eye images in the target area; Performing disparity matching cost calculation on each of the left and right eye images to obtain a plurality of disparity matching cost values; Aggregating the disparity matching cost values ​​in multiple directions, and calculating through a maximum independent variable point set to obtain a disparity map corresponding to a maximum disparity matching cost value; Normalizing the disparity map and performing sub-pixel disparity calculation to obtain a stereo matching result; The disparity matching cost of the left and right images is calculated using the following formula: Among them, x is the grayscale value of the pixel in the left image, y is the grayscale value of the pixel in the right image, μ x is the mean value of x, μ y is the y-mean, is the variance of x, is the variance of y, σ xy is the covariance of x and y, c1 and c2 are constants; The constants c1 and c2 are calculated using the following formula: c1=(k1L) 2 c2=(k2L) 2 Where L is the dynamic range of pixel values, k1 = 0.01, k2 = 0.03; The following formula is used to aggregate the disparity matching cost values ​​in multiple directions: in, represents the aggregation cost value in the propagation direction of r, r represents the propagation direction, C(p, d) represents the matching cost value, represents the aggregation cost of p pixel point on d disparity in the r propagation direction, represents the aggregation cost of the previous pixel of pixel p in the propagation direction r on the disparity d, represents the aggregation cost of the previous pixel of pixel p in the propagation direction r on the d-1 disparity, represents the aggregation cost of the previous pixel of pixel p in the propagation direction r at the disparity of d+1. represents the aggregation cost of the previous pixel of pixel p in the propagation direction r with the minimum aggregation cost among all disparities, and P1 and P2 represent penalty terms.

2. The stereo matching method according to claim 1, characterized in that: Collect multiple consecutive frames of left and right eye images in the target area, and then include: The convolution operator of the deep learning network is used to perform Gaussian filtering calculation on the left and right eye images to extract high-dimensional feature information of the images and improve the accuracy of matching cost calculation.

3. The stereo matching method according to claim 1, characterized in that: The normalizing the disparity map and performing sub-pixel disparity calculation to obtain a stereo matching result specifically includes: Based on the disparity map, a softmax normalization process is performed on each pixel to obtain a normalized result corresponding to the position; Multiply the disparity value of each point by the normalized result of the corresponding position to obtain the sub-pixel disparity value; The sub-pixel disparity value is used as a stereo matching result.

4. A stereo matching system based on a binocular camera, used to implement the method according to any one of claims 1 to 3, characterized in that: The system comprises: An image acquisition unit, used for acquiring multiple consecutive frames of left and right eye images in the target area; A cost value calculation unit, used for performing disparity matching cost calculation on each of the left and right eye images to obtain a plurality of disparity matching cost values; A cost value aggregation unit, configured to aggregate the disparity matching cost values ​​in multiple directions, and calculate the disparity map corresponding to the maximum disparity matching cost value through a maximum independent variable point set; The result output unit is used to normalize the disparity map and perform sub-pixel disparity calculation to obtain a stereo matching result.

5. The stereo matching system according to claim 4, characterized in that: The image acquisition unit is specifically used for: The convolution operator of the deep learning network is used to perform Gaussian filtering calculation on the left and right eye images to extract high-dimensional feature information of the images and improve the accuracy of matching cost calculation.

6. An intelligent terminal, characterized in that: The intelligent terminal includes: a data acquisition device, a processor and a memory; The data acquisition device is used to acquire data; the memory is used to store one or more program instructions; the processor is used to execute one or more program instructions to execute the method according to any one of claims 1-3.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes one or more program instructions, and the one or more program instructions are used to execute the method according to any one of claims 1-3.

Images