Binocular three-dimensional obstacle avoidance method based on adaptive threshold Census transformation

Abstract

The invention discloses a binocular three-dimensional obstacle avoidance method based on adaptive threshold Census transformation, which is suitable for scenes requiring high-depth perception, such as automatic driving obstacle avoidance, robot navigation and the like, and aims to solve the problems that the traditional Census transformation depends on a fixed threshold to be easily interfered by noise and weak texture region feature discrimination is insufficient. The method comprises the following steps: step 1, binocular image acquisition to obtain a left image and a right image respectively; 2, image preprocessing is carried out, and a corrected left image and a corrected right image are generated; 3, cost calculation: respectively carrying out Census transformation on the corrected left image and right image generated in the step 2; step 4, cost aggregation; step 5, parallax calculation and optimization; and step 6, distance measurement and obstacle avoidance. According to the invention, on the basis of the parallel architecture of the FPGA hardware pipeline and the dual-port RAM, the delay of matching cost aggregation and parallax calculation is compressed to a millisecond level, so that the depth perception real-time performance of a dynamic scene is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of FPGA image processing, and in particular to a binocular stereo obstacle avoidance method based on adaptive threshold Census transform, which is applicable to scenarios requiring high-precision depth perception, such as autonomous driving obstacle avoidance and robot navigation. Background Technology

[0002] Stereo matching of binocular images is a core technology for 3D scene reconstruction. Its core task is to recover the depth information of the scene by calculating the disparity of corresponding pixels in the left and right images. For example, Chinese invention patent application CN 120219461A discloses a stereo matching method for binocular images based on disparity distribution ground value modeling. This invention includes the following steps: First, a first stereo matching network is pre-trained using a synthetic dataset to obtain several pre-trained stereo matching networks; then, each binocular image in the synthetic dataset is input into several pre-trained stereo matching networks to obtain several disparity distribution prediction values ​​corresponding to each binocular image; these prediction values ​​are then fused with the disparity ground value map in the Laplacian parameter space and modeled as the corresponding disparity distribution ground value, thereby obtaining a ground value modeling dataset; next, a second stereo matching network is trained using the ground value modeling dataset to obtain a trained second stereo matching network; finally, the binocular image to be predicted is input into the trained second stereo matching network to output the corresponding disparity map. Stereo matching, as the core link in this process, requires accurate localization of corresponding points in the left and right images to calculate the disparity.

[0003] Currently, mainstream algorithms for calculating disparity can be divided into traditional local matching methods and deep learning methods, but both have significant limitations:

[0004] 1. Traditional local matching algorithms perform stably in static scenes, but struggle to meet the real-time requirements of dynamic scenes. Furthermore, the serial computation mode of traditional algorithms cannot fully utilize the parallel architecture of hardware such as FPGAs, resulting in low resource utilization.

[0005] 2. Deep learning methods can significantly improve the reconstruction detail of complex scenes, but they have a "black box" characteristic, which makes parameter adjustment difficult. They also rely on high-performance GPU platforms, making it difficult to achieve low-latency reconstruction in embedded devices.

[0006] The Census transform is a traditional non-parametric region algorithm that preserves the positional information between pixels within a window, thus balancing the impact of illumination information on cost calculation. However, it suffers from over-reliance on the center point, leading to limitations, primarily: 1. Pixel grayscale values ​​are abruptly affected by noise, severely impacting matching accuracy; 2. It fails to fully utilize the inter-pixel relationships, resulting in mismatches in areas with inconsistent depth variations and texture repetition; 3. The Census algorithm is significantly influenced by the center pixel of the neighboring window. Summary of the Invention

[0007] Purpose of the invention: To address the shortcomings of the prior art, this invention provides a binocular stereo obstacle avoidance method based on adaptive threshold Census transform. This invention achieves dynamic noise suppression, establishes a local window grayscale statistical feature model, adaptively adjusts the center pixel judgment threshold, and can utilize FPGA pipeline and dual-port RAM to accelerate algorithm data access, balancing the requirements of algorithm accuracy and matching real-time performance.

[0008] This invention proposes a solution combining adaptive threshold Census transform with hardware acceleration: An adaptive threshold is generated by calculating the mean and standard deviation of local window grayscale values, optimizing the center pixel determination mechanism, and employing 5×5 asymmetric window encoding to improve the distinguishability of weakly textured regions; dual-port RAM is combined to optimize cost space access, compressing the matching latency to milliseconds; semi-global matching and left-right consistency checks are introduced to generate and optimize the disparity map, and median filtering is used to suppress noise. Experiments show that this method achieves a mismatch rate of less than 5% in noisy environments, FPGA throughput reaches 120 FPS (1080P), and the depth calculation error rate is stably controlled within 5%, significantly improving the real-time performance and robustness in autonomous driving scenarios.

[0009] The technical solution is: a binocular stereo obstacle avoidance method based on adaptive threshold Census transform, comprising the following steps:

[0010] Step 1: Binocular image acquisition, obtaining the left and right images respectively;

[0011] Step 2: Image preprocessing to generate the corrected left and right images;

[0012] Step 3, Cost Calculation:

[0013] Perform Census transform on the corrected left and right images generated in step (2) respectively;

[0014] Step 4: Cost aggregation;

[0015] Step 5: Parallax calculation and optimization;

[0016] Step 6: Distance measurement and obstacle avoidance.

[0017] Furthermore, the specific steps of step (1) are as follows:

[0018] Step (11): Using the checkerboard pattern, the binocular camera is calibrated in Matlab using the Zhang Zhengyou calibration method, and the parameters of the left and right cameras of the binocular camera are obtained respectively.

[0019] Step (12): Simultaneously acquire images using a binocular camera, obtaining a left image and a right image. Then, use an FPGA board to display the left and right images correctly on the screen.

[0020] Both the left and right images are RGB color images.

[0021] Furthermore, the left and right camera parameters of the binocular camera in step (11) include intrinsic parameter matrix, extrinsic parameter matrix and distortion coefficient.

[0022] Furthermore, the specific steps of step (2) are as follows:

[0023] Step (21): The left and right images in RGB color form obtained in step (1) are converted by weighted summation to obtain the left and right images in grayscale form, removing color information while retaining image brightness information;

[0024] Step (22): Based on the distortion parameters of the lenses of the left and right cameras of the binocular camera, the distortion of the left and right images is corrected by mathematical transformation to generate the corrected left and right images.

[0025] Furthermore, step (22) includes the following steps:

[0026] Adjust the image coordinates of the left and right images to correct distortions caused by viewing angle or surface curvature, and interpolate new pixel values ​​from the left and right images based on the adjusted image coordinates to generate the corrected left and right images.

[0027] Furthermore, the specific steps of step (3) are as follows:

[0028] (31) Perform improved Census transform on the corrected left and right images generated in step (2) respectively:

[0029] (311) First, use formula (1) to calculate the average gray value I of the pixels within the local m×n window. p ,in:

[0030]

[0031] Among them: I P This represents the grayscale value of pixel p;

[0032] This represents the average grayscale value of pixels within a local m×n window;

[0033] m and n represent the size parameters of the neighborhood window. m and n have the same value and are odd numbers. Preferably, m = n = 5.

[0034] (312) The validity of the center pixel is determined by equation (2) and the adaptive threshold δ, and the optimized grayscale sequences of the corrected left and right images are generated, where:

[0035]

[0036] Where: δ represents the adaptive threshold;

[0037] (32) Using a fixed 5×5 window, the pixel values ​​of the optimized grayscale sequences of the corrected left and right images obtained in step (31) are Census encoded, and two sets of 16-bit binary feature vectors corresponding to the optimized grayscale sequences of the corrected left and right images are output.

[0038] (33) Based on the two sets of 16-bit binary feature vectors obtained in step (32), obtain the Hamming distance corresponding to the feature vector of the corrected left image and the feature vector of the corrected right image;

[0039] (34) Construct a three-dimensional matrix to store the matching cost value of each pixel under each disparity within the disparity range. The size of the cost space value is specifically expressed as Equation (3):

[0040] 1920×1080×(d max -d min +1)(3);

[0041] Where 1920×1080 is the resolution of the acquired image, d max For maximum parallax, d min Minimum parallax;

[0042] (35) Fill the cost space established in step (34) with the Hamming distance (i.e. the cost value calculated by Census transform) between the corrected left image feature vector and the right image feature vector obtained in step (33).

[0043] Furthermore, in step (32), the information of the redundant windows in the central area of ​​the fixed 5×5 window is discarded, and only the pixels of the 16 windows at the edge of the 5×5 window are selected.

[0044] Furthermore, the specific steps of step (4) are as follows:

[0045] Step (41): Use a semi-global matching algorithm with four directions to obtain the minimum value for each direction, where:

[0046] The four directions are 0°, 45°, 90° and 135° respectively;

[0047] The path cost of pixel p along a certain path r is expressed by formula (4):

[0048]

[0049] in:

[0050] Lr(p,d): represents the cumulative path cost of pixel p along path r with parallax d;

[0051] Cost(p,d): represents the initial matching cost of pixel p under disparity d;

[0052] pr: represents the previous neighboring pixel of pixel p on path r;

[0053] Lr(pr,d): The path cost of the previous pixel under the same disparity d;

[0054] Lr(pr,d±1)+p1: The path cost of the previous pixel under disparity d±1 plus the penalty term p1;

[0055] The minimum path cost of the previous pixel among all possible disparities i is added to the penalty term p2;

[0056] The minimum path cost of the previous pixel under all parallaxes;

[0057] Step (42): Averaging the different costs obtained from the four directions according to formula (5) yields the final aggregate cost, and the cost space value is updated, where:

[0058]

[0059] Where: S(p,d) represents the final aggregation cost;

[0060] C ri (p,d) represents the aggregate cost of each path.

[0061] Furthermore, the specific steps of step (5) are as follows:

[0062] Step (51): Obtain the optimal disparity result using the winner-takes-all algorithm.

[0063] The aggregation costs under each disparity are compared, and the disparity value corresponding to the minimum cost is taken as the final disparity result. The calculation formula is shown in Equation (6):

[0064]

[0065] Where, d max For maximum parallax; D p The calculated disparity value;

[0066] Step (52): Using the left image as a reference, perform a consistency check between the left and right images:

[0067] The difference is obtained by subtracting the disparity value d of each pixel in the left and right disparity maps captured by the left and right cameras of the binocular camera. Then, the difference is compared with a preset threshold. If the difference is less than the preset threshold, the disparity of the pixel is determined to be valid; otherwise, it is determined to be an invalid matching point, and the pixel is filled with disparity holes.

[0068] Step (53): Perform median filtering on the obtained image data to further optimize the disparity map quality.

[0069] A 3×3 sliding window is used to sort the neighborhood of each pixel, and the median value is selected to replace the center pixel value.

[0070] Furthermore, the specific steps of step (6) are as follows:

[0071] Step (61): Binocular stereo matching obtains disparity information. The disparity value is disparity = xl - xr. Using the principle of similar triangles, the disparity image is converted using formula (7) to represent the depth information of objects in the scene, that is, the distance information of each pixel is obtained. The formula for calculating the depth information is:

[0072] D= (f*baseline) / disparity (7)

[0073] Where D represents the distance information of the pixel;

[0074] f represents the normalized focal length;

[0075] The baseline is the distance between the optical centers of two cameras, also known as the baseline distance.

[0076] Disparity represents the disparity value;

[0077] Step (62): Calculate the 3D coordinates of each pixel in the camera coordinate system using the depth information obtained in step (61). The calculation formula is shown in (8). Then, obtain the true distance of the target in the world coordinate system through coordinate system transformation. When the distance to the detected obstacle is less than a certain distance, the path planning module generates a detour trajectory, thereby planning a new path to achieve obstacle avoidance.

[0078]

[0079] Where Z represents the depth value;

[0080] D represents parallax;

[0081] X and Y represent spatial coordinate components;

[0082] f represents the camera focal length;

[0083] x 0l y 0l This represents the physical coordinates of the principal point of the camera.

[0084] This invention is a hardware implementation method for binocular stereo obstacle avoidance based on adaptive threshold Census transformation: First, based on the parallel architecture of FPGA hardware pipeline and dual-port RAM, the latency of matching cost aggregation and disparity calculation is compressed to the millisecond level, which significantly improves the real-time performance of depth perception in dynamic scenes.

[0085] Beneficial Effects: The binocular stereo obstacle avoidance method based on adaptive threshold Census transform disclosed in this invention has the following beneficial effects:

[0086] 1. By designing a modular preprocessing workflow (grayscale conversion → distortion correction), the algorithm's dependence on high-performance hardware is reduced, and combined with a memory reuse strategy, resource consumption is reduced by more than 40%.

[0087] 2. By introducing a threshold decision mechanism based on the average gray value of the neighborhood, the interference of sudden changes in the center pixel on the matching results is effectively suppressed, and the impact of illumination fluctuations and random noise on feature encoding is reduced, so that the algorithm can maintain stability and reliability in complex lighting environments.

[0088] 3. The improved Census transform and neighborhood interpolation fusion algorithm doubles the feature discrimination of bit strings in weak texture regions. With the addition of semi-global matching optimization cost aggregation, the average error rate of disparity map is stably controlled within 5%, providing high-precision 3D perception support for real-time obstacle avoidance and path planning of autonomous driving systems. Attached Figure Description

[0089] Figure 1 This is a flowchart of a binocular stereo obstacle avoidance method based on adaptive threshold Census transform disclosed in this invention.

[0090] Figure 2a and Figure 2b This is a schematic diagram of the decomposition process of the Census transformation in Example 1.

[0091] Figure 3 This is a schematic diagram of the three-dimensional cost space in Example 1.

[0092] Figure 4 This is a schematic diagram of the aggregation path for semi-global matching in Example 1.

[0093] Figure 5 This is a schematic diagram of the left-right consistency check process in Example 1.

[0094] Figure 6 This is a schematic diagram of median filtering in Example 1.

[0095] Figure 7 This is a schematic diagram illustrating the principle of disparity map to depth map conversion in Example 1.

[0096] Figure 8a This is the original diagram for Example 2.

[0097] Figure 8b For traditional Census Figure 8a The result after processing.

[0098] Figure 8c This invention discloses a binocular stereo obstacle avoidance method based on adaptive threshold Census transform. Figure 8a The result after processing. Detailed Implementation

[0099] The specific embodiments of the present invention are described in detail below.

[0100] The "range" disclosed in this invention is defined by a lower limit and an upper limit. A given range is defined by selecting a lower limit and an upper limit, which define the boundaries of a particular range. Ranges defined in this way can include or exclude endpoints and can be arbitrarily combined; that is, any lower limit can be combined with any upper limit to form a range. For example, if a range of 10–50 is listed for a specific parameter, it is also expected that ranges of 10–40 and 20–50 are also included. Furthermore, if the minimum range values ​​are 1 and 2, and the maximum range values ​​are 3, 4, and 5, then the following ranges are all expected: 1–3, 1–4, 1–5, 2–3, 2–4, and 2–5. In this application, unless otherwise stated, the numerical range "a–b" represents a shortened representation of any combination of real numbers between a and b, where a and b are real numbers. For example, the numerical range "0–5" means that all real numbers between "0–5" have been listed herein; "0–5" is merely a shortened representation of these numerical combinations.

[0101] Unless otherwise specified, all embodiments and optional embodiments of this application can be combined to form new technical solutions.

[0102] Unless otherwise specified, all technical features and optional technical features of this application may be combined to form new technical solutions.

[0103] Unless otherwise specified, all steps in this application may be performed sequentially or randomly, preferably sequentially. For example, the method includes steps (a) and (b), indicating that the method may include steps (a) and (b) performed sequentially, or it may include steps (b) and (a) performed sequentially. For example, the mention that the method may also include step (c) indicates that step (c) may be added to the method in any order. For example, the method may include steps (a), (b), and (c), or it may include steps (a), (c), and (b), or it may include steps (c), (a), and (b), etc.

[0104] Unless otherwise specified, the terms "comprising" and "including" as used in this application can be open-ended or closed-ended. For example, "comprising" and "including" can mean that other components not listed may also be included, or that only the listed components may be included.

[0105] Unless otherwise specified, the reaction will proceed under normal temperature and pressure conditions.

[0106] Unless otherwise specified, all parts or percentages are by weight or by weight percentage.

[0107] In this invention, all the substances used are known substances that can be purchased or synthesized by known methods.

[0108] In this invention, all the devices or equipment used are conventional devices or equipment known in the art and are readily available.

[0109] Example 1

[0110] A method for stereo obstacle avoidance based on adaptive threshold Census transform includes the following steps:

[0111] Step 1: Binocular image acquisition, obtaining the left and right images respectively;

[0112] Step 2: Image preprocessing to generate the corrected left and right images;

[0113] Step 3, Cost Calculation:

[0114] Perform Census transform on the corrected left and right images generated in step (2) respectively;

[0115] Step 4: Cost aggregation;

[0116] Step 5: Parallax calculation and optimization;

[0117] Step 6: Distance measurement and obstacle avoidance.

[0118] Furthermore, the specific steps of step (1) are as follows:

[0119] Step (11): Using the checkerboard pattern, the binocular camera is calibrated in Matlab using the Zhang Zhengyou calibration method, and the intrinsic parameter matrix, extrinsic parameter matrix and distortion coefficient of the left and right cameras of the binocular camera are obtained respectively.

[0120] Step (12): Simultaneously acquire images using a binocular camera to obtain two images (i.e., the left and right images). Display the left and right images correctly on the screen using the FPGA board.

[0121] Both the left and right images are RGB color images.

[0122] Furthermore, the specific steps of step (2) are as follows:

[0123] Step (21): The left and right images in RGB color form obtained in step (1) are converted by weighted summation to obtain their corresponding grayscale images. Color information is removed while image brightness information is retained to prepare for subsequent feature extraction.

[0124] Step (22): Based on the distortion parameters of the lenses of the left and right cameras of the binocular camera, the image distortion is corrected by mathematical transformation to generate the corrected left and right images.

[0125] Furthermore, step (22) is as follows:

[0126] Adjust the coordinates of the left and right images to correct distortions caused by viewing angle or surface curvature, and interpolate new pixel values ​​from the images based on the adjusted coordinates of the left and right images to generate the corrected left and right images.

[0127] Furthermore, the specific steps of step (3) are as follows:

[0128] Step (31): Perform improved Census transform on the corrected left and right images generated in step (2) respectively.

[0129] (311) Calculate the average gray value I of the pixels within the local m×n window using formula (1). p ;

[0130]

[0131] Among them: I P This represents the grayscale value of pixel p;

[0132] This represents the average grayscale value of pixels within a local m×n window;

[0133] m and n represent the size parameters of the neighborhood window. m and n have the same value and are odd numbers. It is preferable that m = n = 5.

[0134] (312) The validity of the center pixel is determined by equation (2) and the adaptive threshold δ, and an optimized grayscale sequence is generated, where:

[0135]

[0136] Where: δ represents the adaptive threshold;

[0137] Step (32): Using a fixed 5×5 window (discarding redundant information in the central region and selecting only 16 pixels at the edge), Census encoding is performed on the pixel values ​​of the optimized grayscale sequences of the corrected left and right images obtained in step (31), and two sets of 16-bit binary feature vectors corresponding to the optimized grayscale sequences of the corrected left and right images are output.

[0138] The Census transform relies solely on the relative grayscale relationships of pixels within a local window, exhibiting strong robustness to changes in illumination. Compared to traditional matching methods based on grayscale differences, this invention maintains stable matching accuracy even under complex lighting conditions. The traditional Census transform compares the value of the central pixel with its surrounding pixels, assigning a value greater than 0 and a value less than 1 to obtain a feature vector. Using a 5×5 sampling window, redundant information in the central region is discarded, and only 16 pixels at the window edges are selected to generate a 16-bit Census vector, thus obtaining the transformation result. This process is as follows: Figure 2a and Figure 2b As shown.

[0139] Step (33): Based on the two sets of 16-bit binary feature vectors obtained in step (32), obtain the Hamming distance corresponding to the feature vector of the corrected left image and the feature vector of the corrected right image;

[0140] Hamming distance represents the similarity between two neighboring pixel regions. The specific calculation process involves performing an XOR operation on the vectors of the left and right images, constructing a parallel addition tree, and counting the number of 1s to obtain the Hamming distance. This scheme uses a 5×5 sampling window to generate and extract 16 effective sampling points. Through a four-stage pipeline optimization design, the parallel computation of the full-window Hamming distance can be completed in only 4 clock cycles.

[0141] Step (34): Construct a three-dimensional matrix (cost space) to store the matching cost of each pixel under each disparity within the disparity range. The specific value of the cost space is expressed as Equation (3):

[0142] 1920×1080×(d max -d min +1)(3);

[0143] Where 1920×1080 is the resolution d of the acquired image. max For maximum parallax, d min Minimum parallax;

[0144] (35) Fill the view cost space values ​​obtained in step (34) with the Hamming distances (cost values ​​calculated by Census transform) corresponding to the feature vectors of the corrected left image and the corrected right image obtained in step (33), respectively. This process is as follows: Figure 3 As shown, Figure 3 In this context, a equals 1920, and b equals 1080. The cost space is not a single value, but a numerical storage space composed of a group of cost values, which can be abstractly understood as the cost space.

[0145] Furthermore, step (4) is as follows:

[0146] Step (41) uses a semi-global matching algorithm with four directions to obtain the minimum value for each direction, where the four directions are 0°, 45°, 90° and 135° respectively;

[0147] The path cost of pixel p along a certain path r is expressed by formula (4):

[0148]

[0149] in:

[0150] Lr(p,d): represents the cumulative path cost of pixel p along path r with parallax d;

[0151] Cost(p,d): represents the initial matching cost of pixel p under disparity d;

[0152] pr: represents the previous neighboring pixel of pixel p on path r;

[0153] Lr(pr,d): The path cost of the previous pixel under the same disparity d;

[0154] Lr(pr,d±1)+p1: The path cost of the previous pixel under disparity d±1, plus a penalty term p1 (used for small disparity changes);

[0155] The minimum path cost of the previous pixel among all possible disparities i, plus a penalty term p2 (for large disparity changes);

[0156] The minimum path cost of the previous pixel across all disparities is used for numerical normalization to prevent excessive accumulated cost; this process is as follows: Figure 4 As shown.

[0157] Step (42): According to formula (5), the different costs obtained from the four directions are averaged to obtain the final aggregate cost, and the cost space value is updated.

[0158]

[0159] Where: S(p,d) represents the final aggregation cost;

[0160] C ri (p,d) represents the aggregate cost of each path.

[0161] Furthermore, the specific steps of step (5) are as follows:

[0162] Step (51): Obtain the optimal disparity result using the winner-takes-all algorithm.

[0163] The aggregation costs under each disparity are compared, and the disparity value corresponding to the minimum cost is taken as the final disparity result. The calculation formula is shown in Equation (6):

[0164]

[0165] Where, d max For maximum parallax; D p The calculated disparity value;

[0166] Step (52): Using the left image as a reference, perform a consistency check between the left and right images:

[0167] The difference between the image data captured by the left and right cameras of the binocular camera is obtained by subtracting the image data from each camera. This difference is then compared with a preset threshold. If the difference is less than the preset threshold, the left and right images are considered to be consistent; otherwise, they are considered to be inconsistent, and the data set is discarded. This process aims to fill in holes, invalid pixels, and invalid parallax to obtain the optimal parallax map. Figure 5 As shown.

[0168] Step (53): Perform median filtering on the obtained image to further optimize the disparity map quality.

[0169] A 3×3 sliding window is used to sort the neighborhood of each pixel, and the median value is selected to replace the center pixel value. This effectively eliminates isolated noise points and stripe artifacts introduced by dynamic programming. The final optimized disparity map can be directly used for applications such as 3D reconstruction or depth perception. This process is as follows: Figure 6 As shown.

[0170] Furthermore, the specific steps of step (6) are as follows:

[0171] Step (61): Binocular stereo matching obtains disparity information. The disparity value is disparity = xl - xr. The disparity image can be transformed using the principle of similar triangles and formula (7) to represent the depth information of objects in the scene, that is, to obtain the distance information of each pixel. The formula for calculating the depth information is:

[0172] D=(f*baseline) / disparity (7)

[0173] Where D represents the distance information of the pixel;

[0174] f represents the normalized focal length;

[0175] The baseline is the distance between the optical centers of two cameras, also known as the baseline distance.

[0176] Disparity represents the disparity value;

[0177] Step (62): Calculate the 3D coordinates of each pixel in the camera coordinate system using the depth information obtained in step (61). The calculation formula is shown in (8). Then, obtain the true distance of the target in the world coordinate system through coordinate system transformation. When the distance to the detected obstacle is less than a certain distance, the path planning module generates a detour trajectory to achieve obstacle avoidance.

[0178]

[0179] Where Z represents the depth value;

[0180] D represents parallax;

[0181] X and Y represent spatial coordinate components;

[0182] f represents the camera focal length;

[0183] x 0l y 0l This represents the physical coordinates of the camera's principal point. The schematic diagram is as follows: Figure 7 As shown, Figure 7 In the middle, B represents the baseline.

[0184] This technical solution innovatively constructs a stereo vision matching architecture based on a cost aggregation mechanism, effectively addressing the real-time parsing challenge of 3D reconstruction in dynamic scenes. Targeting applications with high timeliness requirements, such as autonomous driving systems and augmented reality navigation, the algorithm achieves low-latency processing performance (sub-millisecond response in typical scenarios) while maintaining sub-pixel-level geometric feature fidelity through heterogeneous computing platform adaptability and computational efficiency optimization. In actual binocular vision measurement, the accuracy of distance measurement typically decreases as the distance between the obstacle and the binocular camera increases, consistent with the basic principle of binocular ranging. To avoid experimental randomness, multiple experiments were conducted, and the relative error remained below 2%.

[0185] Example 2

[0186] The technical solution of Example 1 is used in conjunction with the traditional Census method for processing the original image ( Figure 8a ) to be processed, where:

[0187] Figure 8a This is the original image. Figure 8b For traditional Census Figure 8a The result after processing. Figure 8c This invention discloses a binocular stereo obstacle avoidance method based on adaptive threshold Census transform. Figure 8a The result after processing.

[0188] Experiments showed that the algorithm presented in this application has high matching accuracy, and on three sets of test images, the algorithm had the lowest average mismatch rate of only 5.92%, which is 6.7% higher than the matching accuracy of the traditional Census algorithm. The above comparative experiments demonstrate that the improved algorithm obtained in this paper produces a higher quality disparity map and has more effective matching accuracy when performing stereo matching. The technical solution of this invention can quickly obtain high-precision and dense disparity maps, laying a good foundation for the development of subsequent technologies.

[0189] The embodiments of the present invention have been described in detail above. However, the present invention is not limited to the above embodiments, and various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.

Claims

1. A method for stereo obstacle avoidance based on adaptive threshold Census transform, characterized in that, Includes the following steps: Step 1: Binocular image acquisition, obtaining the left and right images respectively; Step 2: Image preprocessing to generate the corrected left and right images; Step 3, Cost Calculation: Perform Census transform on the corrected left and right images generated in step (2) respectively; Step 4: Cost aggregation; Step 5: Parallax calculation and optimization; Step 6: Distance measurement and obstacle avoidance.

2. The binocular stereo obstacle avoidance method based on adaptive threshold Census transform as described in claim 1, characterized in that, The specific steps of step (1) are as follows: Step (11): Using the checkerboard pattern, the binocular camera is calibrated in Matlab using the Zhang Zhengyou calibration method, and the parameters of the left and right cameras of the binocular camera are obtained respectively. Step (12): Simultaneously acquire images using a binocular camera, obtaining a left image and a right image. Then, use an FPGA board to display the left and right images correctly on the screen. Both the left and right images are RGB color images.

3. The binocular stereo obstacle avoidance method based on adaptive threshold Census transform as described in claim 2, characterized in that, The left and right camera parameters of the binocular camera mentioned in step (11) include intrinsic parameter matrix, extrinsic parameter matrix and distortion coefficient.

4. The binocular stereo obstacle avoidance method based on adaptive threshold Census transform as described in claim 2, characterized in that, The specific steps of step (2) are as follows: Step (21): The left and right images in RGB color form obtained in step (1) are converted by weighted summation to obtain the left and right images in grayscale form, removing color information while retaining image brightness information; Step (22): Based on the distortion parameters of the lenses of the left and right cameras of the binocular camera, the distortion of the left and right images is corrected by mathematical transformation to generate the corrected left and right images.

5. The binocular stereo obstacle avoidance method based on adaptive threshold Census transform as described in claim 4, characterized in that, Step (22) includes the following steps: Adjust the image coordinates of the left and right images to correct distortions caused by viewing angle or surface curvature, and interpolate new pixel values ​​from the left and right images based on the adjusted image coordinates to generate the corrected left and right images.

6. The binocular stereo obstacle avoidance method based on adaptive threshold Census transform as described in claim 1, characterized in that, The specific steps of step (3) are as follows: (31) Perform improved Census transform on the corrected left and right images generated in step (2) respectively: (311) First, use formula (1) to calculate the average gray value of the pixels within the local m×n window. in: Among them: I P This represents the grayscale value of pixel p; This represents the average grayscale value of pixels within a local m×n window; m and n represent the size parameters of the neighborhood window. m and n have the same value and are odd numbers. Preferably, m = n = 5. (312) The validity of the center pixel is determined by equation (2) and the adaptive threshold δ, and the optimized grayscale sequences of the corrected left and right images are generated, where: Where: δ represents the adaptive threshold; (32) Using a fixed 5×5 window, the pixel values ​​of the optimized grayscale sequences of the corrected left and right images obtained in step (31) are Census encoded, and two sets of 16-bit binary feature vectors corresponding to the optimized grayscale sequences of the corrected left and right images are output. (33) Based on the two sets of 16-bit binary feature vectors obtained in step (32), obtain the Hamming distance corresponding to the feature vector of the corrected left image and the feature vector of the corrected right image; (34) Construct a three-dimensional matrix to store the matching cost value of each pixel under each disparity within the disparity range. The size of the cost space value is specifically expressed as Equation (3): 1920×1080×(d max -d min +1)(3); Where 1920×1080 is the resolution of the acquired image, d max For maximum parallax, d min Minimum parallax; (35) Fill the cost space established in step (34) with the Hamming distances corresponding to the corrected left image feature vector and the right image feature vector obtained in step (33).

7. The binocular stereo obstacle avoidance method based on adaptive threshold Census transform as described in claim 6, characterized in that, In step (32), the information of the redundant windows in the central area of ​​the fixed 5×5 window is discarded, and only the pixels of the 16 windows at the edge of the 5×5 window are selected.

8. The binocular stereo obstacle avoidance method based on adaptive threshold Census transform as described in claim 1, characterized in that, The specific steps of step (4) are as follows: Step (41): Use a semi-global matching algorithm with four directions to obtain the minimum value for each direction, where: The four directions are 0°, 45°, 90° and 135° respectively; The path cost of pixel p along a certain path r is expressed by formula (4): in: Lr(p,d): represents the cumulative path cost of pixel p along path r with parallax d; Cost(p,d): represents the initial matching cost of pixel p under disparity d; pr: represents the previous neighboring pixel of pixel p on path r; Lr(pr,d): The path cost of the previous pixel under the same disparity d; Lr(pr,d±1)+p1: The path cost of the previous pixel under disparity d±1 plus the penalty term p1; The minimum path cost of the previous pixel among all possible disparities i is added to the penalty term p2; The minimum path cost of the previous pixel under all parallaxes; Step (42): Averaging the different costs obtained from the four directions according to formula (5) yields the final aggregate cost, and the cost space value is updated, where: Where: S(p,d) represents the final aggregation cost; C ri (p,d) represents the aggregate cost of each path.

9. The binocular stereo obstacle avoidance method based on adaptive threshold Census transform as described in claim 1, characterized in that, The specific steps of step (5) are as follows: Step (51): Obtain the optimal disparity result using the winner-takes-all algorithm. The aggregation costs under each disparity are compared, and the disparity value corresponding to the minimum cost is taken as the final disparity result. The calculation formula is shown in Equation (6): Where, d max For maximum parallax; D p The calculated disparity value; Step (52): Using the left image as a reference, perform a consistency check between the left and right images: The difference is obtained by subtracting the disparity value d of each pixel in the left and right disparity maps captured by the left and right cameras of the binocular camera. Then, the difference is compared with a preset threshold. If the difference is less than the preset threshold, the disparity of the pixel is determined to be valid; otherwise, it is determined to be an invalid matching point, and the pixel is filled with disparity holes. Step (53): Perform median filtering on the obtained image data to further optimize the disparity map quality. A 3×3 sliding window is used to sort the neighborhood of each pixel, and the median value is selected to replace the center pixel value.

10. The method for binocular stereo obstacle avoidance based on adaptive threshold Census transform as described in claim 1, characterized in that, The specific steps of step (6) are as follows: Step (61): Binocular stereo matching obtains disparity information. The disparity value is disparity = xl - xr. Using the principle of similar triangles, the disparity image is converted using formula (7) to represent the depth information of objects in the scene, that is, the distance information of each pixel is obtained. The formula for calculating the depth information is: D=(f*baseline) / disparity (7) Where D represents the distance information of the pixel; f represents the normalized focal length; The baseline is the distance between the optical centers of two cameras, also known as the baseline distance. Disparity represents the disparity value; Step (62): Calculate the 3D coordinates of each pixel in the camera coordinate system using the depth information obtained in step (61). The calculation formula is shown in (8). Then, obtain the true distance of the target in the world coordinate system through coordinate system transformation. When the distance to the detected obstacle is less than a certain distance, the path planning module generates a detour trajectory, thereby planning a new path to achieve obstacle avoidance. Where Z represents the depth value; D represents parallax; X and Y represent spatial coordinate components; f represents the camera focal length; x 0l y 0l This represents the physical coordinates of the principal point of the camera.

Images