Binocular stereo matching FPGA accelerator architecture, method and apparatus

Abstract

The application discloses a binocular stereo matching FPGA accelerator architecture, method and device, belongs to the binocular stereo vision field, and comprises a line buffer module, an initial cost calculation module, a Sobel edge detection module, a cost aggregation module, a right view cost matrix construction module, a uniqueness detection module, a left-right consistency detection module, a hole filling module and a sub-pixel interpolation module. The application solves the problems of low precision, poor efficiency and low resource utilization rate of binocular stereo matching, improves the precision, and guarantees the real-time performance and the efficiency of operation resources.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of binocular stereo vision, and more particularly to a binocular stereo matching FPGA accelerator architecture, method and device. BACKGROUND

[0002] In the process of human cognition of the world, more than 75% of the information is obtained through the human eyes, and the brain processes and responds to the information obtained by the two eyes. Binocular stereo vision simulates the human visual system, acquires two images of the same scene through two cameras at the same time, and calculates the distance and depth information of each object in the scene through the comparison and matching of the two images, so as to realize the perception and understanding of the three-dimensional scene. Stereo matching is widely used in three-dimensional reconstruction, medical imaging, intelligent driving and other fields. By matching images under different angles and finding the deviation of the same point, two-dimensional image to three-dimensional depth reconstruction is completed. Due to light, noise, occlusion, weak texture and real-time requirements, the matching accuracy is easily affected. The matching result directly affects the depth recovery, so that stereo matching becomes the most critical problem in the stereo vision system.

[0003] In the actual matching system, there are still many problems. The current difficulties and key points include low-texture area matching difficulty, occlusion problem, noise and complexity problem, light and color change problem and the like. These problems are mainly caused by the uncertainty and complexity of optical images in texture, light, noise and occlusion. Optimization is needed at each link of stereo matching to improve the accuracy and robustness of the matching result.

[0004] In view of the wide application prospect and many problems to be solved of binocular stereo vision, it is still a great challenge to optimize the difficulties of binocular stereo matching to obtain a matching algorithm with better robustness, and to transplant the algorithm to obtain a hardware acceleration platform balanced in real-time, throughput and solution accuracy. SUMMARY

[0005] The present application aims to overcome the shortcomings of the prior art, and provides a binocular stereo matching FPGA accelerator architecture, method and device, which solves the problems of low precision, poor efficiency and low resource utilization rate of binocular stereo matching, and improves the precision while ensuring real-time performance and high efficiency of operation resources.

[0006] The purpose of the present application is achieved by the following scheme:

[0007] A binocular stereo matching FPGA accelerator architecture comprises:

[0008] The line buffer module, the initial cost calculation module, the Sobel edge detection module, the cost aggregation module, the right view cost matrix construction module, the uniqueness detection module, the left-right consistency detection module, the hole filling module and the sub-pixel interpolation module;

[0009] The line buffer data is sent to the initial cost calculation module and the Sobel edge detection module, the initial matching cost calculation is completed in the initial cost calculation module, and the Sobel edge detection module completes the calculation of the gray edge information; the initial matching cost and the gray edge information enter the cost aggregation module of each path, and the results of the cost aggregation module of each path are summed to obtain the final matching cost;

[0010] The left image integer disparity is obtained through the uniqueness detection module, and the right image integer disparity is obtained by constructing the matching cost matrix of the right image through the right view cost matrix construction module;

[0011] The obtained disparity data is sent to the left-right consistency detection module and the hole filling module for processing, and the final disparity is obtained after sub-pixel interpolation by the sub-pixel interpolation module.

[0012] Further, a cascade FIFO module is further included, after data input, the line buffer is realized through the cascade FIFO, and a 7x7 window is constructed; the cascade FIFO module specifically includes the following running process: at the beginning of each frame, the first row of pixels is written into FIFO 0 according to the row effective signal; the second row of pixels is written into FIFO 0, and the data in FIFO 0 is read out and written into FIFO 1 at the same time; the third row of pixels is written into FIFO 0, and the data in FIFO 0 is read out and written into FIFO 1 at the same time, and the data in FIFO 1 is written into FIFO 2; the process is repeated, and when the seventh row of pixels is input from the port, the 7 rows of pixels are output at the same time.

[0013] Further, the initial cost calculation module includes a Census transformation module and a Hamming distance calculation module; the line buffer data is sent to the Census transformation module, the Census transformation result data is sent to the Hamming distance calculation module through the Census transformation module, and the matching cost calculation is completed through weighting.

[0014] Further, the initial cost calculation module includes an SAD calculation module; the input of the right image of the SAD calculation module is delayed, and then is input into different SAD calculation modules, and then the Hamming distance and the SAD calculation value are weighted to obtain a complete disparity space.

[0015] Further, the Census transformation module is set to module multiplexing, that is, only one Census transformation module is used to buffer the Census transformation result.

[0016] A running method of a binocular stereo matching FPGA accelerator architecture, which is based on the binocular stereo matching FPGA accelerator architecture according to any one of the above, the Sobel edge detection module completes the calculation of gray edge information, comprising the sub-steps of: processing the image by using the Sobel operator, selecting the middle three rows of the row buffer process to combine a 3*3 register array to obtain a 3*3 image window required by the Sobel convolution operation, and selecting the middle three rows of the 7 rows when constructing the register array, performing convolution on the 3*3 image template and the Sobel operator template, the convolution is a multiplication and addition operation between matrices, and the shift operation is used to replace the multiplication of the multiplier; for the signed number addition part, the signed number operation is replaced by the unsigned number operation by using the comparison method.

[0017] Further, the cost aggregation module comprises the following steps: combining the consistency detection and the disparity extraction into one step, and using a cascade bit comparator structure to extract the disparity while completing the uniqueness detection.

[0018] Further, the hole filling module comprises the following steps: for the hole point (x, y), the effective disparity on the left side will be earlier than the target point in the pixel flow order, a first-level register is used for left disparity registration, if the input disparity is not 0, the value of the register is updated, otherwise the original value is kept; for the right disparity, the register operation is performed on the disparity to obtain the effective disparity on the right side.

[0019] Further, the left-right consistency detection module comprises the following steps: selecting the corresponding point disparity within the disparity range, and using an arbitration selector to select for the continuous occlusion area and the mismatch area.

[0020] An apparatus of a binocular stereo matching FPGA accelerator architecture, comprising the binocular stereo matching FPGA accelerator architecture according to any one of the above.

[0021] The beneficial effects of the present application include:

[0022] (1) The application carries out the design of hardware acceleration structure of binocular stereo matching, so as to meet the needs of high precision and real-time in the landing system. The application reserves four paths consistent with the direction of the pixel flow and the R2L direction which can be realized by the row buffer to reduce the delay and complexity of the system. The application uses hardware design ideas such as row buffer, register array, comparator tree, ping-pong operation and arbiter to realize the function, proposes a buffer synchronization structure based on RAM in the cost aggregation module, and proposes a cascaded bit comparator structure to complete the uniqueness detection, finally outputs 14-bit, 8-bit decimal fixed-point dense disparity. Finally, the application realizes the hardware acceleration structure with full pipelining and high parallelism, and the system delay is only 268.74us under the main frequency of 74.25M. The output results are submitted to KITTI 2012 and KITTI 2015 for testing, and the mismatch rate in the non-occluded area is 8.036% and 6.094% respectively, which ranks first in the accuracy of disparity calculation in the stereo vision system.

[0023] (2) The application solves the problems of low precision, poor efficiency and low resource utilization rate of binocular stereo matching, improves the precision, and guarantees the real-time performance and high efficiency of the operation resources. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the drawings needed to be used in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0025] Figure 1 It is a schematic diagram of the overall architecture of the embodiment of the application.

[0026] Figure 2 It is a schematic diagram of the implementation of the cascaded FIFO of the embodiment of the application.

[0027] Figure 3 It is a schematic diagram of the cost calculation module of the embodiment of the application.

[0028] Figure 4 It is a circuit structure diagram of the horizontal direction Sobel operator of the embodiment of the application.

[0029] Figure 5 It is a schematic diagram of the overall architecture of the cost aggregation part of the embodiment of the application.

[0030] Figure 6 It is a schematic diagram of the uniqueness detection disparity extraction module of the embodiment of the application.

[0031] Figure 7 Fig. 1 is a schematic diagram of a tap structure according to an embodiment of the present application;

[0032] Figure 8 Fig. 2 is a schematic diagram of a left-right consistency detection module structure according to an embodiment of the present application;

[0033] Figure 9 Fig. 3 is a schematic diagram of an arbitration plus selector according to an embodiment of the present application;

[0034] Figure 10 Fig. 4 is a schematic diagram of sub-pixel interpolation according to an embodiment of the present application. DETAILED DESCRIPTION

[0035] All features disclosed in this specification, and / or all methods or processes specified in this specification can be combined in any combination, and / or substituted, unless specific exclusion is explicitly provided. In addition, for processes disclosed in this specification that comprise "steps" or "acts" for completion of the processes, the order of the steps or acts of the specification can be changed, unless a specific order is explicitly provided.

[0036] In view of the problems in the background, the inventors of the present application have made creative analyses and reflections, and have found:

[0037] Scharstein conducted an experimental comparison of existing dense stereo matching algorithms, and summarized the traditional stereo matching algorithm, which is divided into four steps of matching cost calculation, matching cost aggregation, disparity extraction and disparity refinement. At the same time, the dense matching algorithm is divided into local stereo matching algorithm and global stereo matching algorithm.

[0038] Local stereo matching (Local Methods) appeared earlier and developed for a long time, has the characteristics of simple calculation and easy hardware implementation, and has good performance in specific scenes. Common local stereo matching algorithms include sum of squared differences (SSD), normalized cross correlation (NCC), sum of absolute differences (SAD), etc. Local stereo matching algorithm mainly calculates the feature information of each pixel point and its adjacent pixel points, has the advantages of small calculation amount, fast calculation speed and low complexity, but is very sensitive to noise, and the matching effect in texture repeated area and occlusion area is not ideal. In view of the shortcomings of local stereo matching, many works have improved it to obtain better matching results. Geiger et al. used multi-scale decomposition to improve the accuracy and robustness of local matching. Hamd et al. proposed a stationary matching probability algorithm based on the previous work, which only uses four-neighbor pixels without specifying window size, and has fast calculation speed, but the edge matching effect is not ideal. Zhu et al. proposed to combine AD, gradient information and Census transform to improve the accuracy and robustness of matching. Zhang et al. proposed NIPM-sWMF algorithm, which uses non-iterative local stereo matching and weighted median filtering to optimize the matching result.

[0039] Unlike local matching methods, global stereo matching algorithms (Global Methods) are to calculate the feature information of the entire image, and the matching problem of corresponding pixel points is transformed into the problem of seeking the global optimum of the matching cost function, so its matching accuracy is generally higher than that of local stereo matching algorithms, and it can adapt to most matching scenes, but the disadvantage is that the computational complexity is high. Commonly used global stereo matching algorithms include energy function-based methods, graph cut-based methods, and global dynamic programming-based methods. Bleyer et al. proposed an algorithm using color segmentation, each segmentation part is described by a plane model, and a mean shift-based clustering algorithm is used to adapt to the global matching cost calculation function. Zitnick et al. proposed a matching method based on image rendering design, which uses over-segmentation of the input image and calculates the matching value of the entire segmentation part instead of individual pixels, and uses loop belief propagation in the MRF framework to calculate the depth of each segmentation part. Yang Qingxiong et al. proposed a minimum spanning tree to complete the cost aggregation process, avoiding the defects of local stereo matching.

[0040] Obviously, global algorithms can achieve higher accuracy, but the resource consumption and computing power consumption they bring are often unacceptable, making it difficult to apply to practical scenarios. In order to solve this problem, in 2005, a new semi-global matching (SGM) algorithm was proposed, which uses mutual information to calculate the matching cost, and its goal is not to optimize iteratively but to minimize the global energy function by aggregating the pixel matching costs in each direction. The final disparity map achieves sub-pixel level accuracy and can detect occlusions. For all standard image sets, the error in the unoccluded area is less than 3%. Semi-global matching opens up new ideas for stereo matching, and compared to local methods, SGM has higher matching accuracy, and compared to global methods, the operation complexity of SGM is greatly reduced. Based on the SGM algorithm, Humenberger proposed an improved SGM algorithm, which segments the image according to the texture strength, and then uses the effective disparity to fit the disparity plane within the segmentation block. Yang Qingxiong et al. proposed the classic AD Census algorithm, which fuses SAD and Census for cost calculation, uses a cross-shaped arm to construct a support window for scan line optimization, and ranks first in the Middleburry test set.

[0041] In summary, the stereo matching problem is the core of stereo vision, and a real-time matching system with high precision and high throughput is a necessity for stereo vision systems. Current stereo matching algorithms and acceleration schemes often use high-end GPUs or FPGAs for deployment, which is relatively expensive, and uses extremely high clock speeds to trade off system throughput, but it is still difficult to balance matching accuracy, matching efficiency, power consumption, and resource area.

[0042] Therefore, the present application aims to solve the problems of low precision, poor efficiency, and low resource utilization of binocular stereo matching, improve precision while ensuring real-time performance and high efficiency of computing resources, and proposes an adaptive semi-global matching scheme (S-SGBM) using a Sobel operator. S-SGBM is a self-defined term of the present application. The present application cooperatively designs a hardware acceleration circuit and deploys it based on Xilinx K7FPGA (but is not limited to this), achieving high-precision, low-latency hardware acceleration circuit design and verification.

[0043] The present application analyzes the problems of poor matching effect of texture regions and edge regions, and insufficient overall robustness of existing algorithms at the algorithm level, and fully considers the friendliness of hardware transplantation, and proposes an adaptive semi-global matching algorithm and its hierarchical strategy. After comparing the acceleration characteristics of CPU, GPU, FPGA and ASIC, in order to achieve wider application scenarios and improve system throughput while ensuring precision, the accurate and real-time FPGA acceleration structure based on two algorithms is cooperatively designed. Specifically, the present application concept includes the following contents:

[0044] 1) In view of the problems of weak anti-noise ability of Census, poor matching effect on repetitive texture and weak texture regions in the cost calculation stage, the SAD cost fusion algorithm is used to calculate the initial disparity in the present application concept.

[0045] 2) In the cost aggregation stage, in view of the problems of edge divergence and unreasonable aggregation area caused by scan line optimization, combined with the ideas of path weighting and support window, and considering the feasibility of hardware transplantation, the adaptive cost aggregation algorithm based on Sobel operator edge detection result is proposed in the present application concept, and the gray value is used as an auxiliary to obtain a better cost aggregation result.

[0046] 3) In the disparity extraction and post-processing stage, the present application concept uses the algorithm process of uniqueness detection, left-right consistency detection, hole filling and sub-pixel interpolation to extract and refine the disparity.

[0047] 4) On the basis of S-SGBM, the present application concept further optimizes the key steps in a hardware-friendly manner, and designs a high-precision and low-latency binocular stereo matching architecture. In the hardware acceleration architecture, a resource-efficient structure of Hamming distance calculation and SAD calculation is proposed for multi-level disparity in the cost calculation step; a ping-pong RAM buffer structure is proposed in the cost aggregation step; and a cascaded bit comparator structure is proposed in the uniqueness detection step.

[0048] As Figure 1As shown, the overall architecture design of the S-SGBM proposed by the application. The overall architecture includes: line buffer module, initial cost calculation module, Sobel edge detection module, five-path cost aggregation module (CostAggr), right view cost matrix construction module, uniqueness detection module (Unique Disp), left-right consistency detection module (LR Check), hole filling module (Occ Fill) and subpixel interpolation module (Subpixel), wherein the initial cost calculation module includes Census transform module, Hamming distance calculation and SAD calculation module.

[0049] The data flow control of the overall architecture is as follows: after data input, first realize line buffering through cascaded FIFO to realize the construction of a 7x7 window; send the line buffered data into the Census transform module, SAD calculation module and Sobel edge detection module, send the Census transform result data into the Hamming distance calculation module, and complete the calculation of the matching cost by weighting. Next, the initial matching cost and the gray edge information enter the aggregation module of each path, and finally the results of each path are summed to obtain the final matching cost, and the left image integer disparity is obtained through the uniqueness detection module. The matching cost matrix of the right image is constructed to obtain the integer disparity of the right image. The disparity data is sent into the left-right consistency detection module and the hole filling module in turn, and the final disparity is obtained after subpixel interpolation.

[0050] The row synchronization and frame synchronization are used to realize the cascaded FIFO. The implementation of the cascaded FIFO is mainly realized by the idea of serial-parallel conversion to output n rows of data at the same time. As shown in Figure 2 At the beginning of each frame, the first row of pixels is written into FIFO 0 according to the row valid signal; the second row of pixels is written into FIFO 0 at the same time, and the data in FIFO 0 is read out and written into FIFO 1; the third row of pixels is written into FIFO 0 at the same time, and the data in FIFO 0 is read out and written into FIFO 1, and the data in FIFO 1 is written into FIFO 2; repeat this process, and when the seventh row is input from the port, 7 rows of pixels are output at the same time. The design points of the cascaded FIFO are summarized as follows: after the beginning of each frame, the write enable of FIFO 0-5 is the row synchronization signal delayed by one row period in turn, and the read enable of the current FIFO is the write enable signal of the next level FIFO, so that when the seventh row arrives, the output of the six-level FIFO is continuous 7 rows of pixels.

[0051] From the perspective of resource efficiency, the application designs different cost construction methods for two different cost calculation methods. First, it is not advisable to use a sliding window for caching. For the 64-level disparity system designed by the application, a 7x7 window requires caching 7x64 pixel data, and in addition, 63 Census transformation modules are required. This is a great consumption of resources. In order to reduce the number of calculations, the application designs a module reuse method, that is, only one Census transformation unit is used, and the cache object is the result of Census transformation. In this way, the number of Census transformation operations is reduced by 98%. The Census transformation vector corresponding to the 64-level disparity and the Census transformation vector of the current pixel of the left image are input into 64 Hamming distance calculation units to obtain all Hamming distance values in the disparity range.

[0052] Unlike the calculation of Hamming distance, which can cache Census vectors, the SAD calculation module requires the subtraction of pixel values of left and right pixels. However, thanks to the design structure proposed by the application, only a 64-level delay of the right image input to the SAD calculation module is required, and the input is respectively input into 64 different SAD calculation modules, without the need for caching the entire window. The Hamming distance and SAD value under 64-level disparity are weighted to obtain the complete disparity space. The cost calculation module is designed, and the complete cost calculation module is as shown in Figure 3 . Figure 3 CTU is a Census transformation unit (Census Transform Unit), HDCU is a Hamming distance calculation unit (Hamming Distance Calculate Unit), and SCU is a SAD value calculation unit (SAD Calculate Unit).

[0053] The cost aggregation step of S-SGBM depends on the edge detection result of the Sobel operator. The Sobel operator processes the image using convolution operation. The middle three rows of the row buffer module are selected to combine a 3x3 register array to obtain the 3x3 image window required for Sobel convolution operation. It should be noted that in order to synchronize the Sobel operation result with the cost calculation aggregation result, the middle three rows of the 7 rows should be selected when constructing the register array, and the 3x3 image template and Sobel operator template are convolved. Convolution is a multiplication and addition operation between matrices. Because the numbers in the Sobel operator template are 0, 1, and 2, only a shift operation is required to replace the use of a multiplier. For the signed number addition part, the application also uses a comparison method to replace the signed number operation with an unsigned number operation, saving bit width and improving resource utilization. Taking the horizontal Sobel operator as an example, as shown in Figure 4The Sobel convolution unit is composed of a shift and a first addition and a first subtraction, and the multiplier-adder is completely abandoned by using the characteristics of the operator template. The size of the results of the first row and the third row is compared after the shift and addition, and the absolute value of the gradient is obtained by unsigned number subtraction. The whole edge detection module is composed of four such convolution units, and there is a difference in the pixel position of operation according to the difference of the Sobel operator in different directions. The whole module is composed of two stages of pipelining, i.e., a first shift and addition and a first subtraction, to shorten the combinational logic link and improve the synthesis performance.

[0054] Figure 5 The overall architecture design of the cost aggregation part is composed of a data buffer structure composed of six true dual-port RAMs, five path cost aggregation units (PCAU) of five paths, and a synchronization structure composed of two pseudo dual-port RAMs. The function of the data buffer is to synchronize the input data of the five cost aggregation modules. For the pixel streams in the four directions of L2R, T2B, LT2RB, and RT2LB, the input is sequential, while the aggregation in the R2L direction needs to be input in reverse, i.e., at least one row of cache is needed for the R2L direction, causing a one-row delay. In the inventive concept, a very common true dual-port RAM (True Dual Port RAM) is used to realize the data buffering and synchronization design before cost aggregation.

[0055] Many works use a comparison tree structure similar to the cost aggregation step to realize the disparity extraction under the WTA algorithm. However, the present invention uses uniqueness detection in the matching step, and although the comparison tree structure can find the minimum value to extract the complete disparity, it is difficult to realize the judgment of the second minimum value, and therefore it is not suitable for the present invention. In the inventive concept, the consistency detection and the disparity extraction are combined into one step, and a cascaded bit comparator structure is proposed to extract the disparity while completing the uniqueness detection. The uniqueness detection and disparity extraction module is shown in Figure 6 The uniqueness detection and disparity extraction module is composed of 10 cascaded bit comparators and 11 stages of pipelining for disparity extraction.

[0056] In stereo matching, the left image is generally set as the reference image, and the right image is the target image. In the present invention, the left image and the right image are also based on this conventional setting. Specifically, the right Figure 1 The matching cost of the point (x, y) at the disparity d is C l (x, y+d, d) in the cost matrix of the left image. The cost aggregation outputs 64 cost aggregation values of 10 bit width, and the value of the point (x, y) at the disparity d=1 in the left image corresponds to C r (x, y-1, 1) in the cost matrix of the right image. According to this rule, d=64 corresponds to C r(x, y-64, 64). Namely, for the disparity d in the right image, the cost aggregation result based on the left image will first output the aggregation value under d = 1. Therefore, the present application gives a tap structure, as shown in Figure 7 All the cost aggregation values are sequentially tapped by 64 taps, and the 64-d tap delay of the disparity d of the left image is taken to re-output the cost aggregation matrix of the right image. After disparity extraction by uniqueness detection, the disparity result of the right image is obtained.

[0057] The left-right consistency detection module structure of the present application is shown in Figure 8 After the disparity of the right image is obtained, the synchronized disparity of the left image and the disparity of the right image are input into the module. When matching with the left image as the reference image, the reference pixel point coordinate is (x, y), and the corresponding disparity d is (x-d, y) in the right image. The 64-level cascade register is used to obtain the right image corresponding disparity of the disparity [1, 64], which is input into a 64-to-1 selector according to the corresponding order of 1 to 64, and the left disparity value is used to select the corresponding right image disparity. After obtaining the disparity of the corresponding position in the right image, comparison is performed. If the difference between them is within 1, it is determined as valid disparity, and the original disparity value is output, otherwise, it is determined as invalid disparity, and the disparity 0 is output.

[0058] In the concept of the present application, the hole filling is to fill the invalid disparity points, and the first valid disparity on the left and right sides of the invalid disparity is used for filling. The valid disparity on the left side is better extracted, because for the hole point (x, y), the valid disparity on the left side will precede the target point in the pixel stream order, and a first-level register is used for left disparity registration. If the input disparity is not 0, the value of the register is updated, otherwise, the original value is maintained. For the right side disparity, since it is later than the target hole in the pixel stream order, the disparity needs to be registered to obtain the valid disparity on the right side.

[0059] Considering the characteristics of left-right consistency detection, the detection method is to select the corresponding point disparity within the disparity range. For continuous occluded regions and mismatched regions, most of the holes are smaller than the disparity range D. Therefore, in order to extract the right side disparity, the present application designs a scheme of arbitration plus selector, as shown in Figure 9The arbitration is used to locate the position of the first valid disparity on the right side, and the selector completes the work of extracting the disparity value. The disparity is sequentially registered using a similar 64-stage cascade register, and the output of the 64th stage is the synchronized disparity and is used as the input for extracting the valid disparity on the left side. After obtaining the 64th-stage right disparity, it is determined whether the disparity is not 0, and a one-dimensional vector is formed by splicing from low to high according to the disparity order from 1 to 64. At this time, the disparity selection problem is converted into a common problem in the design of a polling arbitrator: how to find the lowest 1. As shown in the following formula, Grant is the final finding result, and Req is the one-dimensional vector that needs to be determined, the output of which is 1 for the lowest 1 and 0 for the other bits. If the input is 10010, the output is 00010. After obtaining the position of the first valid disparity on the right side, a 64-to-1 selector is used to complete the extraction of the valid disparity on the right side.

[0060] Grant=(~Req+1)&Req 4

[0061] The sub-pixel interpolation in the inventive concept needs a synchronized aggregated cost value as input, and the module structure is as shown in Figure 10 Since the disparity is delayed by 64 taps in the hole filling module, a FIFO is needed to synchronize the aggregated cost. The sub-pixel interpolation formula used in the present application is shown in the following formula, and the calculation of the value of p in the following formula is needed:

[0062] d=d int +d subpix

[0063]

[0064] In the above formula, d int represents the integer-pixel disparity, and d subpix represents the sub-pixel disparity, which is between -0.5 and 0.5. represents the matching cost corresponding to d int . and represent the matching costs corresponding to the two disparities adjacent to the optimal disparity. The specific operation process of the function f is as follows: the difference between the matching costs of the adjacent disparity and the optimal disparity is:

[0065]

[0066] The specific calculation method of the sub-pixel is as follows:

[0067]

[0068] Using three 64-to-1 selectors, respectively, the current disparity value d, d-1, d+1 is input, the corresponding cost of the current disparity and the adjacent disparity is extracted, and then the difference value is obtained. The divider IP is used to complete the division to obtain p, and the multiplier IP is used to obtain p 2 Since p is less than 1, considering the resource condition and the operation unit efficiency, the fixed-point integer bit 1 and the decimal bit 6 of the divider output are set, and the same fixed-point precision of the multiplier is set. The final output of the sub-pixel is 1 bit and the value is 0, and the fixed-point of the decimal bit is 8 bits, that is, the fixed-point precision is 1 / 256, about 0.0039.

[0069] In summary, a perfect binocular vision system can be widely used in robot, assisted driving, three-dimensional modeling, augmented reality and other application scenarios. Through the technical scheme based on the application, binocular stereo matching can be realized, depth perception and reconstruction of three-dimensional scenes can be realized, and more accurate information can be provided for the application of these fields. In addition, the research of binocular stereo vision also has important significance for the research of human visual perception and cognitive mechanism. The process of researching binocular stereo matching can help us better understand the mechanism of human visual perception and further explore the working principle of human brain. Combined with the achievements of life science and computer science, the binocular stereo matching technology is used to provide help in navigation and obstacle detection through depth perception and environmental understanding, to provide more convenience for the rehabilitation and life of visually impaired people, and to help them better adapt to life and work.

[0070] It should be noted that the following embodiments can be combined and / or extended, replaced, in any logical manner within the scope of protection defined in the claims of the application, for example, disclosed technical principles, disclosed technical features or implied disclosed technical features.

[0071] Embodiment 1

[0072] A binocular stereo matching FPGA accelerator architecture, comprising:

[0073] A row buffer module, an initial cost calculation module, a Sobel edge detection module, a cost aggregation module, a right view cost matrix construction module, a uniqueness detection module, a left-right consistency detection module, a hole filling module and a sub-pixel interpolation module;

[0074] The row buffer data is sent to the initial cost calculation module and the Sobel edge detection module. The initial cost calculation module completes the calculation of the initial matching cost, and the Sobel edge detection module completes the calculation of the gray edge information. The initial matching cost and the gray edge information enter the cost aggregation module of each path, and the results of the cost aggregation module of each path are summed to obtain the final matching cost.

[0075] The left integer disparity is obtained by the uniqueness detection module, and the right integer disparity is obtained by constructing the matching cost matrix of the right image using the right image cost matrix construction module;

[0076] The obtained disparity data is sent to the left-right consistency detection module and the hole filling module for processing, and the final disparity is obtained after sub-pixel interpolation by the sub-pixel interpolation module.

[0077] Embodiment 2

[0078] On the basis of embodiment 1, a cascade FIFO module is further included, and after data input, row buffering is realized by the cascade FIFO to realize the construction of a 7x7 window; the cascade FIFO module specifically includes the following running process: at the beginning of each frame, the first row of pixels is written into FIFO 0 according to the row valid signal; the second row of pixels is written into FIFO 0 at the same time, and the data in FIFO 0 is read out and written into FIFO 1; the third row of pixels is written into FIFO 0 at the same time, and the data in FIFO 0 is read out and written into FIFO 1, and the data in FIFO 1 is written into FIFO 2; repeat the process, and when the seventh row is input from the port, the pixels of the seven rows are output at the same time.

[0079] Embodiment 3

[0080] On the basis of embodiment 1, the initial cost calculation module includes a Census transformation module and a Hamming distance calculation module; the row buffering data is sent to the Census transformation module, the Census transformation result data is sent to the Hamming distance calculation module by using the Census transformation module, and the matching cost calculation is completed by weighting.

[0081] Embodiment 4

[0082] On the basis of embodiment 1, the initial cost calculation module includes an SAD calculation module; the input of the right image of the SAD calculation module is delayed and then input to different SAD calculation modules, and then the Hamming distance and the SAD calculation value are weighted to obtain a complete disparity space.

[0083] Embodiment 5

[0084] On the basis of embodiment 3, the Census transformation module is set to module multiplexing, that is, only one Census transformation module is used to buffer the Census transformation result.

[0085] Embodiment 6

[0086] A running method of a binocular stereo matching FPGA accelerator architecture, which is based on the binocular stereo matching FPGA accelerator architecture described in any one of embodiments 1-5, and the Sobel edge detection module completes the calculation of gray edge information, including the following sub-steps: processing the image by using the Sobel operator, selecting the middle three rows of the row buffer process to combine a 3*3 register array to obtain a 3*3 image window required by the Sobel convolution operation, and selecting the middle three rows of the 7 rows when constructing the register array, performing convolution on the 3*3 image template and the Sobel operator template, the convolution is a multiplication and addition operation between matrices, and the shift operation is used to replace the multiplier; for the signed number addition part, the signed number operation is replaced by the unsigned number operation by using the comparison method.

[0087] Embodiment 7

[0088] On the basis of embodiment 6, the cost aggregation module includes the following steps: combining the consistency detection and the disparity extraction into one step, and using a cascade bit comparator structure to extract the disparity while completing the uniqueness detection.

[0089] Embodiment 8

[0090] On the basis of embodiment 6, the hole filling module includes the following steps: for the hole point (x, y), the valid disparity on the left side will be earlier than the target point in the pixel flow order, and a first-level register is used for left disparity registration, if the input disparity is not 0, the value of the register is updated, otherwise the original value is kept; for the right disparity, the register operation is performed on the disparity to obtain the valid disparity on the right side.

[0091] Embodiment 9

[0092] On the basis of embodiment 6, the left-right consistency detection module includes the following steps: selecting the corresponding point disparity within the disparity range, and using an arbitration selector to select for the continuous occlusion area and the mismatch area.

[0093] Embodiment 10

[0094] A binocular stereo matching FPGA accelerator architecture device includes the binocular stereo matching FPGA accelerator architecture described in any one of embodiments 1-5.

[0095] The units described in the embodiments of the present application can be implemented in the form of software or hardware, and the described units can also be arranged in a processor. In some cases, the names of these units do not constitute a limitation on the units themselves.

[0096] According to an aspect of an embodiment of the present application, there is provided a computer program product or computer program, comprising computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the method provided in the various optional implementation manners described above.

[0097] As another aspect, the embodiments of the present application also provide a computer readable medium, which can be included in the electronic device described in the above embodiments, or can exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to implement the method described in the above embodiments. The parts of the present application not involved are the same as or can be implemented by using the prior art.

Claims

1. An apparatus of a binocular stereo matching FPGA accelerator architecture, the apparatus comprising: Comprise: Line buffer module, initial cost calculation module, Sobel edge detection module, cost aggregation module, right view cost matrix construction module, uniqueness detection module, left and right consistency detection module, hole filling module and sub-pixel interpolation module; The line buffer data is sent to the initial cost calculation module and the Sobel edge detection module, and the initial matching cost calculation is completed in the initial cost calculation module, and the Sobel edge detection module completes the gray edge information calculation; the initial matching cost and the gray edge information enter the cost aggregation module of each path, and the results of the cost aggregation module of each path are summed to obtain the final matching cost; The left image integer disparity is obtained through the uniqueness detection module, and the right image matching cost matrix is constructed through the right view cost matrix construction module to obtain the right image integer disparity; The obtained disparity data is sent to the left and right consistency detection module and the hole filling module for processing, and the final disparity is obtained after sub-pixel interpolation by the sub-pixel interpolation module. The initial cost calculation module comprises a Census transformation module, a Hamming distance calculation module and an SAD calculation module; the line buffer data is sent to the Census transformation module, the Census transformation result data is sent to the Hamming distance calculation module by using the Census transformation module; the input of the SAD calculation module right image is delayed and then input to different SAD calculation modules; the results output by the Hamming distance calculation module and the results output by the SAD calculation module are weighted together to complete the initial matching cost calculation.

2. The binocular stereo matching FPGA accelerator architecture apparatus of claim 1, wherein, The line buffer module specifically comprises a cascade FIFO module, which realizes line buffering through the cascade FIFO after data input, and realizes the construction of a 7x7 window; the cascade FIFO specifically comprises the following running process: at the beginning of each frame, the first row of pixels is written into FIFO 0 according to the row effective signal; the second row of pixels is written into FIFO 0 while the data in FIFO 0 is read out and written into FIFO 1; the third row of pixels is written into FIFO 0 while the data in FIFO 0 is read out and written into FIFO 1, and the data in FIFO 1 is written into FIFO 2; repeat the process until the seventh row of pixels is input from the port, and the pixels of seven rows are output at the same time.

3. The device of binocular stereo matching FPGA accelerator architecture according to claim 1, wherein, The Census transformation module is set to module multiplexing, that is, only one Census transformation module is used to buffer the Census transformation result.

4. A method for operating a binocular stereo matching FPGA accelerator architecture, the method comprising: The method is based on the binocular stereo matching FPGA accelerator architecture device of claim 2, and the Sobel edge detection module completes gray edge information calculation, including the following sub-steps: processing an image by using a Sobel operator, selecting three middle rows of a row buffer process to combine a 3*3 register array to obtain a 3*3 image window required by Sobel convolution operation, and selecting three middle rows of seven rows when constructing the register array, performing convolution on the 3*3 image template and the Sobel operator template, the convolution is a multiplication and addition operation between matrices, and a shift operation is used to replace a multiplier utility; for the signed number addition part, the signed number operation is replaced by an unsigned number operation by using a comparison method.

5. The method of claim 4, wherein, The consistency detection and the disparity extraction are combined into one step, and a cascade bit comparator structure is used to complete the uniqueness detection and extract the disparity.

6. The method of claim 4, wherein, The hole filling module runs the following steps: for a hole point (x, y), the valid disparity on the left side precedes the target point in the pixel flow order, a first-level register is used for left-side disparity registration, if the input disparity is not 0, the value of the register is updated, otherwise the original value is kept; for the right-side disparity, a registration operation is performed on the disparity to obtain the valid right-side disparity.

7. The method of Claim 4, wherein, In the disparity range, the corresponding point disparity is selected, and for the continuous occlusion area and the mismatch area, an arbitration and selector mode is used for selection.

Images