Binocular stereo matching hardware architecture supporting variable disparity range

Abstract

The application discloses a binocular stereo matching hardware architecture supporting variable disparity range, which can realize real-time switching of disparity working range in work by configuring a cost calculation and aggregation module through a disparity range switching controller without changing algorithm processing details, can switch the disparity range worked by hardware according to different application scenarios, performs different disparity range calculation in different working scenarios, and supports a working environment requiring a large disparity range while the hardware can work in a small disparity range mode in an open outdoor scene, so that the calculation amount is greatly reduced, and the working power consumption of the overall hardware is reduced.

Description

Technical Field

[0001] This invention relates to the field of stereo vision technology, and in particular to a binocular stereo matching hardware architecture that supports variable parallax range. Background Technology

[0002] In recent years, computer vision technology has developed rapidly, and stereo vision, as an important branch of computer vision, has become one of the most active research areas. Stereo vision uses two-dimensional images obtained by a camera in a scene to reconstruct the three-dimensional information of objects. Based on the number of visual sensors used in the system, it can be divided into three main categories: monocular vision, binocular vision, and multi-view vision. Among them, binocular vision can directly simulate the way human eyes process objects, which is reliable and convenient. Using related algorithms of binocular vision, the depth information of objects in binocular images can be obtained, thus enabling its widespread application in various embedded systems, such as robotics, 3D scene reconstruction, drone navigation, and autonomous driving.

[0003] To achieve a high-precision, low-resource, and low-power real-time binocular stereo matching embedded system, many researchers have focused on optimizing the hardware architecture of binocular stereo matching to achieve low resource and low power consumption. This is achieved by simplifying the algorithm or hardware architecture. However, this method will more or less cause some loss of accuracy and is not suitable for different scenarios. Summary of the Invention

[0004] The purpose of this invention is to provide a binocular stereo matching hardware architecture that supports variable parallax range. Without changing the details of the algorithm processing, the hardware can switch the parallax range it operates in according to different application scenarios. It can perform different parallax range calculations for different working scenarios. While supporting working environments that require a large parallax range, the hardware can also use a smaller parallax range working mode in open outdoor scenarios, which greatly reduces the amount of computation and thus reduces the overall power consumption of the hardware.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] A binocular stereo matching hardware architecture supporting variable disparity range includes: an inter-frame detector, a disparity range switching controller, a filtering module, a cost calculation and aggregation module, a disparity selection module, a sub-pixel level disparity estimation module, and a post-processing module; wherein:

[0007] The inter-frame detector is used to detect the frame blanking period of the video stream data and output a corresponding enable signal to the parallax range switching controller.

[0008] The disparity range switching controller determines whether to switch the disparity range working mode of the binocular stereo matching hardware architecture based on the enable signal, and configures the cost calculation and aggregation module and the disparity selection module.

[0009] The filtering module is used to denoise the input left and right images;

[0010] The cost calculation and aggregation module is used to calculate the path cost based on the denoised left and right images and perform weighted aggregation to obtain the aggregated cost value of each pixel.

[0011] The disparity selection module is used to generate an initial disparity map based on the aggregated cost of each pixel;

[0012] The subpixel disparity estimation module is used to fit the initial disparity map to obtain a subpixel disparity map.

[0013] The post-processing module is used to perform noise reduction processing on the sub-pixel level disparity map to obtain the final disparity map.

[0014] As can be seen from the technical solution provided by the present invention, by configuring the cost calculation and aggregation module through the parallax range switching controller, the parallax working range can be switched in real time during operation. The parallax working range can be switched in different application scenarios, reducing a lot of calculation in specific scenarios, and achieving lower hardware power consumption of the overall system while ensuring accuracy and hardware resources are basically not increased. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A schematic diagram of a binocular stereo matching hardware architecture supporting variable parallax range provided in an embodiment of the present invention;

[0017] Figure 2 A flowchart of a variable disparity range binocular stereo matching based on a semi-global stereo matching algorithm provided in an embodiment of the present invention;

[0018] Figure 3 This is a schematic diagram of the structure of the initial cost calculation module provided in an embodiment of the present invention;

[0019] Figure 4 This is a schematic diagram of the path aggregation cost calculation module provided in an embodiment of the present invention;

[0020] Figure 5 This is a schematic diagram of the path cost aggregation module structure provided in an embodiment of the present invention;

[0021] Figure 6 This is a schematic diagram of all path aggregation costs generated in two cycles for the path aggregation cost provided in an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0023] First, the following explanations are provided for the terms that may be used in this article:

[0024] The term "and / or" means that either or both can be achieved simultaneously. For example, X and / or Y means that it includes both "X" or "Y" as well as the three cases of "X and Y".

[0025] The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0026] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0027] Unless otherwise explicitly specified or limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this document according to the specific circumstances.

[0028] The following is a detailed description of a binocular stereo matching hardware architecture supporting variable parallax range provided by the present invention. Contents not described in detail in the embodiments of the present invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of the present invention, they shall be performed according to conventional conditions in the art or conditions recommended by the manufacturer. Where the manufacturers of the instruments used in the embodiments of the present invention are not specified, they are all conventional products that can be purchased commercially.

[0029] This invention provides a binocular stereo matching hardware architecture that supports variable parallax range, such as... Figure 1 As shown, it mainly includes: an inter-frame detector, a disparity range switching controller, a filtering module, a cost calculation and aggregation module, a disparity selection module, a sub-pixel level disparity estimation module, and a post-processing module; among which:

[0030] The inter-frame detector is used to detect the frame blanking period of the video stream data and output a corresponding enable signal to the parallax range switching controller. Specifically, the input of the inter-frame detector is the video stream synchronization signal. By detecting and counting the edges of the video stream synchronization signal, it is determined whether the video stream data is in the frame blanking period. If so, an enable signal to increase the parallax range is output.

[0031] The disparity range switching controller determines whether to switch the disparity range operating mode of the binocular stereo matching hardware architecture based on the enable signal, and configures the cost calculation and aggregation module and the disparity selection module. Specifically, the disparity range switching controller includes: a disparity range switching enable judgment unit, which determines whether to switch the disparity range operating mode of the binocular stereo matching hardware architecture based on the enable signal and the externally input disparity range selection signal; and a status configuration register group, used to configure the parameters or status of the control registers or selectors of the cost calculation and aggregation module.

[0032] The filtering module is used to denoise the input left and right images; for example, it uses Gaussian filtering to denoise the images and improve the stereo matching accuracy.

[0033] The cost calculation and aggregation module is used to calculate the path cost based on the denoised left and right images and perform weighted aggregation to obtain the aggregated cost value of each pixel.

[0034] The disparity selection module is used to generate an initial disparity map based on the aggregated cost value of each pixel. Specifically, the disparity selection module selects the minimum cost value of each pixel within the working disparity range based on the aggregated cost value of each pixel to obtain the initial disparity map. The disparity selection module includes a multi-level numerical comparator, used to select the minimum cost value of each pixel within the working disparity range based on the aggregated cost value of each pixel.

[0035] The subpixel-level disparity estimation module is used to fit the initial disparity map to obtain a subpixel-level disparity map. Specifically, the subpixel-level disparity estimation module transforms the initial disparity map of integer level into a subpixel-level disparity map with decimal disparity values ​​by using a parabolic fitting method. The subpixel-level disparity estimation module includes two dividers for calculating the decimal disparity values.

[0036] The post-processing module is used to perform noise reduction processing on the sub-pixel level disparity map to obtain the final disparity map.

[0037] Figure 1 In the example shown, the disparity range selection signal can be either 64 or 128 (these two disparity ranges are just examples). The disparity range switching controller determines whether to switch the disparity range working mode of the binocular stereo matching hardware architecture based on the input disparity range selection signal and the enable signal output by the inter-frame detector. Figure 1 In the diagram, CLK52M and CLK104M represent the clock domains in which the circuit module operates within the range indicated by the dashed line.

[0038] In this embodiment of the invention, the cost calculation and aggregation module includes: an initial cost calculation module, a path cost calculation module, and a path cost aggregation module; wherein: the initial cost calculation module is used to calculate the initial cost of pixel matching degree based on Census transform from the denoised left and right images, and obtain the initial cost value of each pixel under the working disparity range; the path cost calculation module is used to calculate the cost value of each pixel under different paths under the working disparity range based on the initial cost value output by the initial cost calculation module using the SGM (semi-global stereo matching) binocular stereo matching algorithm; the path cost aggregation module is used to perform weighted summation of the path cost values ​​of each pixel under different paths under the working disparity range based on the cost values ​​of each pixel under different paths under the working disparity range output by the path cost calculation module, and obtain the aggregated cost value of each pixel under the working disparity range.

[0039] In this embodiment of the invention, the initial cost calculation module includes: multiple (e.g., 2*5) row cache RAMs (random access memory), two sets of shift registers, and an XOR calculation array. The multiple row cache RAMs and the two sets of shift registers are used together to obtain the initial cost calculation vector, and the XOR calculation array is responsible for calculating the initial cost value. The initial cost value is calculated every other column for the leftmost and rightmost multi-column (e.g., 256 columns) regions of the image.

[0040] In this embodiment of the invention, the path cost calculation module includes: multiple (e.g., 5) true dual-port RAMs, multiple (e.g., 4) FIFO row storage units, and multiple addition trees and comparators forming a path cost calculation unit, used to calculate the cost value of each pixel point under different paths within the working parallax range, solving the problem of data dependency asynchrony caused by different data flow directions under different paths and calculating the path cost value under different paths.

[0041] In this embodiment of the invention, the path cost aggregation module includes: a two-stage adder for weighted summation of each path cost value to obtain the corresponding aggregated cost value, and adding it to the corresponding timing control unit.

[0042] To more clearly demonstrate the technical solution and its effects provided by the present invention, the hardware architecture provided by the embodiments of the present invention will be described in detail below with reference to specific examples.

[0043] I. Overall Hardware Implementation Scheme.

[0044] In this embodiment of the invention, a hardware implementation scheme for variable disparity range is provided. The system clock is doubled during semi-global binocular stereo matching processing to obtain a time window with double the margin compared to the original. A disparity range switching controller is added to configure control parameters for relevant modules between frames. Taking 64 and 128 disparity ranges as examples, the scheme of this invention can achieve the goal of calculating the path cost of a disparity range of 128 by time-division multiplexing a hardware computing module with a disparity range of 64, with very few added control modules. The main process is as follows: Figure 2 As shown, it includes:

[0045] Step 1: Set the parallax working range according to the work scenario (for example, the 64 or 128 parallax range mentioned above).

[0046] Step 2: The inter-frame detector detects the frame blanking period and outputs a parallax range change enable signal.

[0047] Step 3: The parallax range switching controller receives the parallax range change enable signal and configures the initial cost calculation module, path cost calculation module, path cost aggregation module, and parallax selection module in conjunction with the set parallax working range.

[0048] For example: Suppose the parallax range is switched from 64 to 128 (or from 128 to 64; of course, other parallax ranges can also be selected as needed). In step 3 above, the parallax range switching controller receives the enable signal for the change of parallax range from 64 to 128, and the set parallax working range = 128. The parallax range switching controller will configure the multiplexer (MUX) selection signals of the left and right Census sequences in the initial cost calculation module, such as... Figure 3 As shown, specify the range of the output sequence for the left and right graphs, specifically: Figure 3 The middle arrow describes the data bit width. The left image selects 24 bits from 2*24 bits for output (i.e., output sequence 2-to-1), and the right image selects 64*24 bits from 130*24 bits for output (i.e., output sequence 130-to-64). To reduce computation and on-chip cache resources without significantly affecting accuracy, edge skipping is performed on both images when calculating the initial cost. Specifically, for the leftmost and rightmost 256 columns of data (edge ​​information is not the focus of this invention), the initial cost is calculated every other row. That is, for the left image, the output sequence is 2-to-1, and for the right image, it is 130-to-64. Since the initial cost calculation only requires one clock cycle, and the hardware performs clock doubling for the semi-global binocular stereo matching process, 64 initial cost calculation units can complete the initial cost calculation for the 64 / 128 disparity range in two cycles. The disparity range switching controller will configure the path cost calculation control register (config) in the path cost calculation module, such as... Figure 4 As shown, the path cost calculation module controls when to operate and when to output. In the first cycle, it calculates the path cost for the first 64 disparity ranges. If the disparity range is 64, the circuit state remains unchanged in the second cycle, without any switching, reducing dynamic power consumption. If the disparity range is 128, it calculates the path cost for the remaining 64 disparity ranges. The disparity range switching controller will configure the path cost aggregation module's accumulation unit to output control signals, such as... Figure 5 As shown, the CTR0 and CTR1 control signals of the MUX (multiplexer) in the accumulation unit of the path cost aggregation module control the output period of the path cost aggregation module. REG (Register) is a register used in the timing circuit to temporarily store signal values; as shown... Figure 6 As shown, the path aggregation cost module generates the aggregated cost for all paths within a 128 disparity range in two cycles. Specifically: the first cycle uses the first-level adder ( Figure 5 The first-level adder in the first cycle calculates the aggregation cost for the disparity range of 0 to 63. The second cycle uses the second-level adder. Figure 5 The second-level adder in the first-level adder calculated the aggregation cost for the disparity range of 64 to 127. Figure 5 The first-stage adder calculates the aggregate cost for the disparity range of 0-63 for the next pixel, forming a pipeline. The aggregate cost for the disparity range of 0-127 for each pixel is calculated in two cycles. The disparity range switching controller configures the disparity selection module calculation controller to control the data flow of the numerical comparison unit. The minimum value of 128 disparity costs is calculated in two cycles, or the minimum value of 64 disparity costs is obtained in one cycle, thus obtaining the initial disparity map.

[0049] II. Description of each part of the hardware architecture.

[0050] 1. Inter-frame detector.

[0051] like Figure 1 As shown, the inter-frame detector determines whether the current state is in the inter-frame blanking period based on synchronization signals such as the frame valid signal of the input video stream. If so, it raises the parallax range to change the enable signal; otherwise, it keeps it at a low level.

[0052] 2. Parallax range switching controller.

[0053] like Figure 1 As shown, after receiving the disparity range changeable enable signal from the inter-frame detector, the disparity range switching controller configures the parameters of the control registers of the initial cost calculation module, the path cost calculation module, and the disparity selection module according to the pre-input disparity range.

[0054] Again, taking the 64 and 128 parallax ranges mentioned earlier as an example, the parallax range switching controller will configure the multiplexer selection signals of the left and right Census sequences in the initial cost calculation module, such as... Figure 3 As shown, the range of the output sequences for the left and right images is specified (64 / 128). To reduce computational load and on-chip cache resources without significantly affecting accuracy, edge skipping is applied to the left and right images when calculating the initial cost. That is, for the leftmost and rightmost 256 columns of data in the image, the initial cost is calculated for every other row. In the skipped column region, a 2-to-1 selection is made for the output sequence of the left image, and a 130-to-64 selection is made for the output sequence of the right image. Since the initial cost calculation can be completed in only one clock cycle, and the clock frequency is doubled for semi-global stereo matching processing, 64 initial cost calculation units can complete the initial cost calculation for the 64 / 128 disparity range in two cycles. When calculating the initial cost for the 64 disparity range, after completing all initial cost calculations in the first cycle, the result is temporarily stored in a register. In the second cycle, the data is retrieved from the register, padded with 0s at the low bits, and passed to the next stage through a multiplexer. When calculating the initial cost for a 128-parallax range, the first cycle completes the calculation of the initial cost for the first 64 parallax ranges and temporarily stores the results in a register. The second cycle completes the calculation of the initial cost for the remaining 64 parallax ranges and merges the result with the result in the register for the first 64 parallax ranges to form the initial cost result for the 128-parallax range, which is then sent to the next stage via a multiplexer. The parallax range switching controller configures the path cost calculation control register in the path cost calculation module, specifying when the path cost calculation module operates and when it outputs, such as... Figure 4As shown, the parallax range switching controller will be configured with four path cost calculation modules. In the first cycle, the path cost for the first 64 parallax ranges is calculated. If the parallax range is 64, the circuit state remains unchanged in the second cycle without any switching. If the parallax range is 128, the path cost for the remaining 64 parallax ranges is calculated. The parallax range switching controller will be configured with a path cost aggregation module, and the accumulation unit will output a control signal, such as... Figure 5 As shown, the CTR0 and CTR1 control signals of the multiplexer in the accumulation unit of the path cost aggregation module control the output cycle of the path cost aggregation module. The parallax range switching controller will configure the parallax selection module to calculate the minimum value of 128 parallax costs in two cycles (or obtain the minimum value of 64 parallax costs in one cycle and output the minimum value to the next level in the second cycle).

[0055] 3. Filtering module.

[0056] In this embodiment of the invention, the filtering module uses Gaussian filtering to denoise the left and right images. For example, a 3x3 Gaussian filtering window can be used to denoise the input left and right images, thereby improving the accuracy of the subsequent semi-global stereo matching algorithm.

[0057] 4. Initial cost calculation module.

[0058] In this embodiment of the invention, an initial cost calculation module based on the Census transform is used. For example, a 5x5 sliding window can be used to obtain a 24-bit 0 / 1 sequence based on the relationship between the surrounding pixel values ​​and the center pixel value, and the number of 1s in the sequence is used as the initial cost value. The Census transform function formula is as follows:

[0059]

[0060]

[0061] in, It is the center pixel within the window. It is a non-center pixel within the window. This represents the pixel obtained after XOR summation. The initial generation value, It represents the pixel values ​​of all non-center pixels in the window. It is the pixel value of the center pixel of the window. It refers to all non-center pixels in the window. Represents a 24-bit sequence. This represents a bitwise XOR operation between sequences. By taking the 24-bit sequences of the left and right images and performing an XOR operation, a new sequence string is obtained. The number of 1s in the sequence string is the initial value of the current pixel under the current disparity. The smaller the initial value, the closer the information in the two windows is, that is, the closer the information between the two pixels is, and vice versa.

[0062] The hardware architecture of the initial cost calculation module is as follows: Figure 3 As shown, each of the left and right images uses 5 BRAMs (line cache RAMs) as line caches to generate a 5*5 sliding window and generates a 24-bit 0 / 1 sequence through comparison operations. The left image uses 2 shift registers to store this sequence and a 2-to-1 multiplexer to select the sequence to be calculated in the image edge skip region for initial cost calculation. The right image uses 130 shift registers to store the 0 / 1 sequence and a 130-to-64 multiplexer to select 64 sequences to be calculated in the image edge skip region for initial cost calculation. The initial low price corresponding to the left and right images is calculated using a set of XOR calculation arrays (…). Figure 3 The initial cost (calc) within the dashed box is calculated based on the Census transform. After calculating the initial cost for the first 64 disparity ranges in one cycle, the result is temporarily stored in a register. If only the initial cost for the 64 disparity ranges is calculated, the result is retrieved in the second cycle, with the lower bits padded with 0s to match the data line width, and sent to the next stage via a 2-to-1 multiplexer. If the initial cost for the 128 disparity ranges is calculated, the initial cost result for the last 64 disparity ranges is obtained after the second cycle, and this result is combined with the result for the first 64 disparity ranges in the register before outputting.

[0063] 5. Path cost calculation module.

[0064] In this embodiment of the invention, the SGM algorithm path cost calculation module is used. SGM needs to calculate the path cost of each pixel under different paths. In this embodiment, four paths are selected to calculate the path cost of the pixel: left to right, top left to bottom right, top to bottom, and right to left. The path cost function formula for each direction is as follows:

[0065] , -

[0066] in, Indicates along the path At pixel Parallax is The path cost at that time. For pixels At parallax The initial value of time. Indicates along the path At pixel The parallax of the previous pixel is The path and value of time. Indicates along the path At pixel The parallax of the previous pixel is The path and value of time. Indicates along the path At pixel The minimum cost value for all parallaxes of the previous pixel. , These are the relevant penalty coefficients, used to improve the matching accuracy of the algorithm. The hardware architecture of the path cost calculation module is as follows: Figure 4 As shown, two true dual-port RAMs are used as row buffers to cache the initial cost and grayscale image, respectively. The two true dual-port RAMs simultaneously read data from low to high and output data from high to low addresses (i.e., for a row of data, output from left to right and from right to left simultaneously to obtain a reverse data stream for calculating the path cost of the RL path). For the TB and LTB paths, FIFO0 caches a row of grayscale image data and outputs it to synchronize the TB and LTB path data. RAM4, FIFO1, FIFO2, and FIFO3 are used as row buffers to cache the path costs of each path and send the results to the corresponding path cost calculation unit (PCCU) to resolve data dependencies in the path cost calculation. At the same time, the results are sent to the path cost aggregation module (PCAU) for aggregate cost calculation. Each path cost calculation unit consists of multiple addition trees and comparators (not shown in the figure). The LR path represents a left-to-right path, the RL path represents a right-to-left path, the TB path represents a top-to-bottom path, and the LTB path represents a top-left to bottom-right path.

[0067] 6. Path cost aggregation.

[0068] Path cost aggregation involves weighted summation of the path costs of the four paths from left to right, from top left to bottom right, from top to bottom, and from right to left in the previous step. The entire summation process is divided into two stages: the first stage calculates the path aggregation cost for the disparity range of 0-63, and the second stage calculates the path aggregation cost for the disparity range of 64-127. If the current operating mode is a disparity range of 64, the path aggregation cost calculation result is temporarily stored in the output register for one clock cycle in the second stage. In both operating modes, the final path aggregation cost is output after two clock cycles. The hardware architecture of the path cost aggregation module is as follows: Figure 5 As shown, it includes two stages of adders to perform weighted summation of the cost values ​​of each path to obtain the corresponding aggregate cost value. The two leftmost ADDs form the first-stage adder, and the single ADD to its right forms the second-stage adder. Corresponding timing control signals (i.e., ...) are also added. Figure 5 The path cost of the CTR0 and CTR1 signals in the signal is weighted and summed to satisfy the two parallax ranges of 64 and 128.

[0069] 7. Parallax selection module.

[0070] Disparity selection involves comparing the 64 / 128 aggregated cost values ​​corresponding to each pixel generated in the previous stage, and selecting the disparity corresponding to the minimum aggregated cost value for each pixel as its final disparity value. Since the path aggregation cost generated in the previous stage is generated over two clock cycles, disparity selection will also take two clock cycles to select the minimum cost value and its corresponding disparity value.

[0071] 8. Subpixel-level disparity estimation module.

[0072] Subpixel-level disparity estimation involves fitting integer disparities to disparity values ​​with decimal parts. Since the disparity values ​​obtained in previous stages through column differences between pixels were all integers, but the depth information of objects in real-world scenes rarely changes in integer units, it is necessary to fit integer disparities to disparity values ​​with decimal parts. This makes the obtained disparity values ​​more accurate and the resulting disparity map smoother. A parabolic fitting method is used for this purpose.

[0073] 9. Post-processing module.

[0074] Post-processing further eliminates noise and smooths the disparity map through methods such as median filtering and bad pixel filling. Median filtering uses a sliding window to determine the number of disparity values ​​of 0 within the window; if the number exceeds a certain threshold, the disparity value at the center point of the window is set to the median of the non-zero disparity values ​​within the window. Bad pixel filling sets the disparity value of 0 points to the nearest smallest non-zero disparity value.

[0075] Based on the hardware architecture described above, the following is a brief description of their collaborative workflow, using the 64 and 128 disparity ranges as examples: The inter-frame detector detects the frame blanking period and raises the disparity range change enable signal. At this time, the disparity range switching controller starts working. It configures the parameters of the control registers and multiplexers of the initial cost calculation module, path cost calculation module, path cost aggregation module, and disparity selection module for the externally input 64 / 128 disparity range working mode. After the configuration is completed, it waits for a new frame of data to enter the variable disparity range binocular stereo matching hardware. The main processing steps of this invention are divided into multiple steps. The first step is to perform Gaussian filtering on the input left and right images, and the output result is directly fed to the initial cost calculation module to calculate the initial cost under the disparity range of each pixel. After obtaining the initial cost, the path cost calculation module will calculate the path cost of multiple paths under the disparity range of each pixel based on the initial cost. The path cost aggregation uses a weighted sum of multiple path costs to obtain the aggregated cost under the disparity range of each pixel. Subsequently, the disparity selection module will select the disparity value corresponding to the minimum cost among all aggregated costs as the disparity value of that pixel, thereby obtaining the initial disparity map. The initial disparity map enters the sub-pixel level disparity estimation module to estimate the fractional part of the disparity value, obtaining a disparity map formed by a more accurate disparity value with a fractional part. Finally, the disparity map is post-processed to obtain a dense and accurate output disparity map.

[0076] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A binocular stereo matching hardware architecture supporting variable parallax range, characterized in that, include: The system comprises an inter-frame detector, a disparity range switching controller, a filtering module, a cost calculation and aggregation module, a disparity selection module, a sub-pixel level disparity estimation module, and a post-processing module; among which: The inter-frame detector is used to detect the frame blanking period of the video stream data and output a corresponding enable signal to the parallax range switching controller; wherein, the input of the inter-frame detector is the video stream synchronization signal, and by detecting and counting the edge of the video stream synchronization signal, it is determined whether the video stream data is in the frame blanking period; if so, an enable signal to increase the parallax range is output. The disparity range switching controller determines whether to switch the disparity range working mode of the binocular stereo matching hardware architecture based on the enable signal, and configures the cost calculation and aggregation module and the disparity selection module. The filtering module is used to denoise the input left and right images; The cost calculation and aggregation module is used to calculate the path cost based on the denoised left and right images and perform weighted aggregation to obtain the aggregated cost value of each pixel. The disparity selection module is used to generate an initial disparity map based on the aggregated cost of each pixel; The subpixel disparity estimation module is used to fit the initial disparity map to obtain a subpixel disparity map. The post-processing module is used to perform noise reduction processing on the sub-pixel level disparity map to obtain the final disparity map.

2. The binocular stereo matching hardware architecture supporting variable parallax range according to claim 1, characterized in that, The cost calculation and aggregation module includes: an initial cost calculation module, a path cost calculation module, and a path cost aggregation module; wherein: The initial cost calculation module is used to calculate the initial cost of the pixel matching degree based on Census transform from the left and right images after denoising, and obtain the initial cost value of each pixel in the working disparity range. The path cost calculation module is used to calculate the cost of each pixel point under different paths within the working disparity range based on the initial cost value output by the initial cost calculation module and the SGM binocular stereo matching algorithm. The path cost aggregation module is used to perform a weighted summation of the path cost values ​​of each pixel under the working parallax range based on the cost values ​​of different paths for each pixel output by the path cost calculation module, so as to obtain the aggregated cost value of each pixel under the working parallax range.

3. The binocular stereo matching hardware architecture supporting variable parallax range according to claim 2, characterized in that, The initial cost calculation module includes: multiple row cache RAMs, two sets of shift registers, and a set of XOR calculation arrays. The multiple row cache RAMs and the two sets of shift registers are used together to obtain the initial cost calculation vector, and the set of XOR calculation arrays is responsible for calculating the initial cost value; wherein the initial cost value is calculated every other column for the leftmost and rightmost multi-column regions of the image.

4. The binocular stereo matching hardware architecture supporting variable parallax range according to claim 2, characterized in that, The path cost calculation module includes: multiple true dual-port RAMs, multiple FIFO row storage units connected in sequence, and multiple path cost calculation units, used to calculate the cost of each pixel for different paths within the working parallax range.

5. The binocular stereo matching hardware architecture supporting variable parallax range according to claim 2, characterized in that, The path cost aggregation module includes a two-stage adder for weighted summation of each path cost value to obtain the corresponding aggregated cost value.

6. The binocular stereo matching hardware architecture supporting variable parallax range according to claim 1, characterized in that, The parallax range switching controller includes: The parallax range switching enable judgment unit combines the enable signal and the externally input parallax range selection signal to determine whether to switch the parallax range working mode of the binocular stereo matching hardware architecture. The status configuration register group is used to configure the parameters or status of the control registers or strobes of the cost calculation and aggregation module.

7. The binocular stereo matching hardware architecture supporting variable parallax range according to claim 1, characterized in that, The disparity selection module selects the minimum cost value of each pixel within the working disparity range based on the aggregate cost value of each pixel, and obtains the initial disparity map. The disparity selection module includes a multi-level numerical comparator, used to select the minimum cost of each pixel within the working disparity range based on the aggregate cost of each pixel.

8. The binocular stereo matching hardware architecture supporting variable parallax range according to claim 1, characterized in that, The subpixel-level disparity estimation module transforms the initial disparity map of integer level into a subpixel-level disparity map with fractional disparity values ​​by using a parabolic fitting method. The subpixel-level disparity estimation module includes two dividers for calculating disparity values ​​with decimals.

Images