Mobile robot binocular vision global positioning system and method

Abstract

The application provides a mobile robot binocular vision global positioning system and method, relates to the technical field of robot vision navigation, and extracts light-independent edge contour feature points by performing adaptive exposure compensation on binocular images of left and right cameras, combines with parallax calculation to generate a high-quality depth feature map, simultaneously constructs a feature unit according to an inflection point, and thus the global position of the mobile robot is determined. In this way, the problem that the global positioning accuracy is reduced due to unstable vision feature extraction caused by sharp changes in light intensity in the multi-region switching scene of the mobile robot can be solved to a certain extent.

Description

Technical Field

[0001] This invention relates to the field of robot vision navigation technology, and more specifically, to a binocular vision global positioning system and method for mobile robots. Background Technology

[0002] With the rapid development of mobile robot technology, visual positioning systems, as one of the core technologies for robot autonomous navigation and environmental perception, have received widespread attention. Binocular vision technology, due to its ability to calculate depth information from images from left and right cameras, offers advantages such as low cost and high accuracy, and has been widely used in mobile robot localization. Through binocular vision systems, robots can acquire depth information of the 3D environment and, combined with pose estimation algorithms, calculate their own position and orientation. However, in practical applications, the performance of binocular vision systems is often limited by lighting conditions, scene complexity, and feature extraction algorithms. For example, in scenes with drastic lighting changes or insufficient ambient light, image quality degrades significantly, leading to inaccurate depth information calculations. Furthermore, when dealing with complex environments (such as dynamic scenes or sparsely textured areas), binocular vision localization systems often struggle to extract stable feature points, thus affecting subsequent localization accuracy. Therefore, improving the robustness and accuracy of binocular vision systems in complex scenes has become a current research hotspot in the field.

[0003] While existing binocular vision localization methods have solved basic localization problems to some extent, they still face several technical bottlenecks. First, the impact of lighting variations on image quality is difficult to eliminate. Existing technologies often employ fixed exposure parameters or simple image enhancement algorithms, failing to adapt to different lighting conditions, resulting in low accuracy in feature extraction and depth calculation. Second, traditional feature extraction methods underutilize edge contour information and cannot effectively combine object geometric features for scene matching, making the localization results susceptible to noise interference. Furthermore, in the global localization process, existing technologies largely rely on direct point cloud matching or 2D image matching, ignoring the spatial correlation and geometric constraints between features, making it difficult to achieve high-precision localization in complex scenes. In summary, existing technologies have shortcomings in lighting adaptability, feature representation ability, and global localization accuracy, limiting the application effectiveness of binocular vision localization systems in real-world scenarios. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention is proposed. This invention provides a binocular vision global localization system and method for mobile robots. It can, to some extent, solve the problem of decreased global localization accuracy caused by unstable visual feature extraction due to drastic changes in light intensity in multi-area switching scenarios (such as from indoors to outdoors, or from bright areas to dim areas) for mobile robots.

[0005] According to one aspect of the present invention, a binocular vision global localization system for a mobile robot is provided, comprising:

[0006] The data acquisition module is used to acquire binocular images from the left and right cameras of the mobile robot, and to obtain the pose information and illumination sensor data of the mobile robot, wherein the pose information includes the three-dimensional spatial coordinates and attitude angle data of the mobile robot.

[0007] An adaptive exposure compensation module is used to perform adaptive exposure compensation on the binocular image based on the light sensor data, extract edge contour feature points in the image, calculate the depth information of the edge contour feature points based on parallax, and generate a depth feature map after light compensation.

[0008] The feature unit extraction module is used to identify the inflection point position of the object edge contour in the depth feature map and take the edge contour segment between two adjacent inflection points as feature units.

[0009] The contour feature construction module is used to calculate the geometric shape parameters and relative depth difference of the feature unit and construct a lighting-independent contour feature descriptor.

[0010] The global localization module is used to perform scene matching using the contour feature descriptor, and combined with the pose information, to determine the global position of the mobile robot using a graph structure optimization algorithm.

[0011] Furthermore, the adaptive exposure compensation module performs exposure compensation on the binocular image, including a lighting environment analysis and projection calculation stage, an adaptive exposure compensation stage, a directional edge feature extraction stage, and a stereo matching and depth calculation stage.

[0012] Furthermore, in the lighting environment analysis and projection calculation stage, the scene's main lighting direction vector is calculated using a weighted vector synthesis method. and overall light intensity I L And the calculated main illumination direction vector The image plane is transformed using the camera projection matrix to obtain a two-dimensional illumination projection vector.

[0013] Furthermore, in the adaptive exposure compensation stage, for each pixel (x,y) in the binocular image, the adaptive exposure compensation coefficient α(x,y) is calculated by combining its local gradient information and illumination projection information.

[0014] When the local gradient direction is perpendicular to the illumination direction, the compensation coefficient reaches its maximum value, and the compensated image pixel value is calculated.

[0015] The adaptive exposure compensation coefficient α(x,y) is calculated as follows:

[0016]

[0017] Where α(x,y) is the compensation coefficient at pixel (x,y), and its value ranges from [0,1]. Let be the gradient vector at pixel (x, y). Let be the projection vector of the illumination direction onto the image plane. Let be the magnitude of the gradient vector. Let β(x,y) be the magnitude of the illumination projection vector, and let β(x,y) be the adaptive gain parameter.

[0018] Furthermore, the directional edge feature extraction stage extracts edges from the compensated image and calculates the gradient operator M(x,y);

[0019] The Harris response value R(x,y) is calculated on the detected edge contours to locate feature points;

[0020] Based on the calculated gradient operator value M(x,y) and Harris response value R(x,y), a feature point reliability metric S(x,y) is constructed, and a reliability metric S(x,y) greater than an adaptive threshold T is selected. s The points are used as candidate feature points;

[0021] Let the set of feature points obtained from the selection be denoted as P = p i |S(p i )>T s , used for the stereo matching process in the stereo matching and depth calculation stage.

[0022] Furthermore, the feature unit extraction module identifies inflection points including:

[0023] The points to be analyzed are selected along the edge contour of the depth feature map, and local curve segments are formed by the forward and backward pixels of each point to be analyzed.

[0024] The coordinates of the contour points of the local curve segment are converted into parametric equations, and the local curve equations are reconstructed through cubic spline interpolation.

[0025] The tangent vectors of the point to be analyzed and its forward and backward pixels are calculated based on the local curve equation, and potential inflection points are marked by the change in the angle between the tangent vectors.

[0026] Calculate the local curvature value of potential inflection points. When the local curvature value shows a changing trend, calculate the curvature acceleration and select the potential inflection point with the largest curvature acceleration as the inflection point.

[0027] Calculate the angle between the depth gradient direction and the contour tangent vector at the inflection point. When the angle is within a preset range, mark the inflection point as a high-confidence inflection point, and optimize the retention of high-confidence inflection points based on the curvature change value and depth difference.

[0028] Furthermore, the illumination-independent contour feature descriptor is obtained by calculating feature parameters and normalizing the sampling point sequence based on the feature unit.

[0029] The feature parameters include the arc length, principal direction, curvature feature, depth gradient of the sampling point, and the correlation function of the sampling point sequence.

[0030] Furthermore, the global positioning module determines the global position of the mobile robot by including:

[0031] A spatiotemporal feature map is constructed, along with two types of core constraints; these two types of core constraints include feature matching constraints and pose continuity constraints.

[0032] The two types of core constraints are transformed into optimization problems. Through iterative optimization, the pose estimation at each time step is adjusted to ensure that the overall constraints are best satisfied.

[0033] The optimized pose sequence is fused with the previously saved trajectory to obtain the robot's position in the global coordinate system.

[0034] Furthermore, the feature matching constraint is used to describe the spatial relationship of the same contour features at different times, and the pose continuity constraint ensures the smoothness of the robot's motion; by performing weighted iterative optimization on these two types of constraints until the pose estimation converges, an accurate position result is obtained.

[0035] According to another aspect of the present invention, a binocular vision global localization method for a mobile robot is provided, comprising:

[0036] The system acquires binocular images from the left and right cameras of the mobile robot and obtains the pose information and illumination sensor data of the mobile robot, wherein the pose information includes the three-dimensional spatial coordinates and attitude angle data of the mobile robot.

[0037] Adaptive exposure compensation is performed on the binocular image based on the illumination sensor data, edge contour feature points in the image are extracted, and depth information of the edge contour feature points is calculated based on parallax to generate an illumination-compensated depth feature map.

[0038] In the depth feature map, the inflection point position of the object edge contour is identified, the edge contour segment between two adjacent inflection points is taken as the feature unit, the geometric shape parameters and relative depth difference of the feature unit are calculated, and a light-independent contour feature descriptor is constructed.

[0039] The contour feature descriptor is used for scene matching, and combined with the pose information, a graph structure optimization algorithm is used to determine the global position of the mobile robot.

[0040] Compared with existing technologies, the binocular vision global localization system and method for mobile robots provided by this invention adaptively compensates for exposure in the binocular images from the left and right cameras, extracts illumination-independent edge contour feature points, and generates a high-quality depth feature map by combining parallax calculation. Simultaneously, feature units are constructed based on inflection points to determine the global position of the mobile robot. This can, to some extent, solve the problem of unstable visual feature extraction and decreased global localization accuracy caused by drastic changes in illumination intensity in multi-area switching scenarios. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0042] Figure 1 This is a system block diagram of a binocular vision global localization system for a mobile robot according to an embodiment of the present invention;

[0043] Figure 2 This is a flowchart of a binocular vision global localization method for a mobile robot according to an embodiment of the present invention. Detailed Implementation

[0044] Hereinafter, exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments of the present invention. It should be understood that the present invention is not limited to the exemplary embodiments described herein.

[0045] Figure 1 This is a system block diagram of a binocular vision global localization system for a mobile robot according to an embodiment of the present invention. Figure 1 As shown, the binocular vision global localization system for a mobile robot includes:

[0046] The data acquisition module is used to acquire binocular images from the left and right cameras of the mobile robot, and to obtain the pose information and illumination sensor data of the mobile robot, wherein the pose information includes the three-dimensional spatial coordinates and attitude angle data of the mobile robot.

[0047] An adaptive exposure compensation module is used to perform adaptive exposure compensation on the binocular image based on the light sensor data, extract edge contour feature points in the image, calculate the depth information of the edge contour feature points based on parallax, and generate a depth feature map after light compensation.

[0048] The feature unit extraction module is used to identify the inflection point position of the object edge contour in the depth feature map and take the edge contour segment between two adjacent inflection points as feature units.

[0049] The contour feature construction module is used to calculate the geometric shape parameters and relative depth difference of the feature unit and construct a lighting-independent contour feature descriptor.

[0050] The global localization module is used to perform scene matching using the contour feature descriptor, and combined with the pose information, to determine the global position of the mobile robot using a graph structure optimization algorithm.

[0051] The data acquisition module utilizes a binocular camera system installed at the front of the mobile robot. The left and right cameras are 12 cm apart, with their optical axes parallel. The cameras have a field of view of 85 degrees, a resolution of 1920×1080 pixels, and a sampling frequency of 30 Hz, simultaneously acquiring RGB color images. The mobile robot's pose information is obtained through a high-precision fiber optic gyroscope, including three-dimensional position coordinates (X, Y, Z) and Euler angles (Roll, Pitch, Yaw) in the global coordinate system, with a sampling frequency of 200 Hz. The illumination sensor data is acquired by an ambient light sensor array located on the top of the mobile robot. This array includes eight evenly distributed illumination sensors with a measurement range of 0-100,000 lux and a sampling frequency of 50 Hz, used to acquire the ambient light intensity distribution information around the robot in a 360-degree radius.

[0052] The adaptive exposure compensation module performs exposure compensation on the binocular image, including the illumination environment analysis and projection calculation stage, the adaptive exposure compensation stage, the directional edge feature extraction stage, and the stereo matching and depth calculation stage.

[0053] Phase 1: Illumination Environment Analysis and Projection Calculation

[0054] Based on an ambient light sensor array mounted on top of the mobile robot, light intensity data L from eight directions is acquired. i (i = 1, 2, ..., 8). The scene's main illumination direction vector is calculated using a weighted vector synthesis method. and overall light intensity I L :

[0055]

[0056] Wherein, the weighting coefficient w i The sensor's value is dynamically adjusted based on its operating status and reliability; the smaller the fluctuation in the sensor's value, the greater its weight. This is a normalized direction vector, representing the orientation of each sensor. The calculated main illumination direction vector... The image plane is transformed using the camera projection matrix to obtain a two-dimensional illumination projection vector. This vector is used in subsequent image processing.

[0057] The second stage, adaptive exposure compensation:

[0058] For each pixel (x, y) in the binocular image, an adaptive exposure compensation coefficient α(x, y) is calculated by combining its local gradient information and illumination projection information. The compensation coefficient is calculated using a modified sigmoid function, which considers the relationship between the pixel gradient direction and the illumination direction.

[0059]

[0060] Wherein, the adaptive gain parameter β(x,y) is calculated using the following formula:

[0061]

[0062] Where k1 and k2 are empirical adjustment coefficients, determined through extensive experiments, with typical values ​​of 0.8 and 0.3, respectively. max This represents the maximum range value of the light sensor. a(x,y) is the compensation coefficient at pixel (x,y), with a value range of [0,1]. Let be the gradient vector at pixel (x, y). Let be the projection vector of the illumination direction onto the image plane. Let be the magnitude of the gradient vector. Let be the magnitude of the light projection vector.

[0063] The compensation coefficient reaches its maximum value when the local gradient direction is perpendicular to the illumination direction, which enhances structural edges that are unaffected by the illumination direction. The compensated image pixel values ​​are calculated using the following formula:

[0064] I ′ (x,y)=I(x,y)·(1+α(x,y))

[0065] Among them, I ′ (x,y) represents the compensated pixel value, ranging from [0,255], I(x,y) represents the original pixel value, ranging from [0,255], and α(x,y) represents the compensation coefficient.

[0066] It should be noted that the key to this stage lies in the calculation of the adaptive gain parameter, which is related to the angle between the gradient direction and the illumination direction. The compensation effect is strongest when the gradient direction is perpendicular to the illumination direction.

[0067] The third stage is the extraction of directional edge features:

[0068] An improved Canny operator is applied to the compensated image to extract edges. First, the gradient operator M(x,y) considering the illumination direction is calculated:

[0069]

[0070] Where γ is the directional enhancement coefficient, with a value range of [0.2, 0.5], and θ(x, y) is the local gradient direction angle. L G is the direction angle of the light projection vector. x G represents the gradient component in the x-direction. y This represents the gradient component in the y-direction. This improvement results in higher response values ​​at edges perpendicular to the illumination direction.

[0071] On the detected edge contours, the improved Harris response value R(x,y) is calculated to locate feature points:

[0072] R(x,y)=det(H)-λ·trace 2 (H)·ω(x,y)

[0073] Where ω(x,y) is the illumination adaptive weight:

[0074]

[0075] Where H is the Harris matrix, a 2×2 matrix, det(H) is the determinant of the Harris matrix, and trace 2 (H) is the square of the trace of the Harris matrix, λ is the tradeoff parameter with a typical value of 0.04, and ω(x,y) is the illumination adaptive weight.

[0076] Based on the calculated gradient operator value M(x,y) and Harris response value R(x,y), a feature point reliability metric S(x,y) is constructed, wherein the reliability metric S(x,y) is the product of the gradient operator value M(x,y) and the Harris response value R(x,y). A reliability metric S(x,y) greater than an adaptive threshold T is selected. s Points are selected as candidate feature points, and the set of selected feature points is denoted as P = p i |S(p i )>T s This set of feature points will be used in the subsequent stereo matching process.

[0077] It should be noted that the adaptive threshold T s The reliability metric S(x,y) is set based on the obtained reliability metric values ​​of all pixels. The mean (S(x,y)) and standard deviation (std(S(x,y)) of the reliability metric over the entire image region are statistically calculated, and weighting coefficients are set accordingly. The mean reflects the baseline level of the overall reliability of feature points in the current scene, while the standard deviation characterizes the dispersion of the reliability distribution. When scene illumination changes drastically, the values ​​of the two weighting coefficients are dynamically adjusted by calculating the illumination change rate of the image in real time (based on the time derivative of the illumination sensor data).

[0078] The fourth stage, stereo matching and depth calculation:

[0079] When matching feature points detected in the left and right images, a matching cost function that takes into account the effect of illumination is introduced:

[0080]

[0081] The matching weight φ(x,y) is calculated as follows:

[0082]

[0083] Where d is the disparity value in pixels, W is the matching window, typically 11×11 pixels in size, and I′ l (x,y) represents the pixel values ​​of the left image after compensation, I′ r (xd,y) represents the pixel value of the right image after compensation, φ(x,y) represents the matching weight, and S(x,y) represents the reliability metric.

[0084] This weighting function ensures that in regions with drastic lighting changes, the matching process relies more on structural information than brightness information. Finally, the depth Z of the feature points is calculated based on the obtained disparity values:

[0085]

[0086] Where f is the camera focal length, typically 6mm, and B is the binocular baseline length, set to 120mm. Through the above calculations, a depth feature map is finally generated, containing the 3D coordinates (X,Y,Z) of feature points and the edge response intensity M(x,y).

[0087] Through multiple interconnected computational steps, adaptive compensation for illumination changes and robust feature extraction are achieved. From illumination environment perception, image compensation, feature extraction to stereo matching, each step fully considers the impact of illumination changes, and improves the reliability of feature extraction through strategies such as adaptive weighting and directionality enhancement. The algorithm's innovation lies mainly in the calculation method of the adaptive illumination compensation coefficient and the feature extraction and matching strategy that considers illumination direction. These improvements significantly enhance the system's performance in complex illumination environments.

[0088] For example, when a mobile robot enters a naturally lit glass doorway from an indoor corridor, the sudden change in light intensity often leads to overexposure in bright areas or loss of features in dark areas using traditional feature extraction methods. The adaptive lighting compensation mechanism in this solution effectively addresses this situation: when the robot's ambient light sensor detects a sudden increase in light intensity (e.g., from 1000 lux to 50000 lux), the algorithm automatically increases the compensation coefficient in dark areas while decreasing it in bright areas. Specifically, for structural edges perpendicular to the lighting direction, such as the doorway frame, the compensation coefficient α(x,y) approaches 1 because its gradient direction is perpendicular to the lighting direction, thus maintaining the clarity of these important structural features. For areas parallel to the lighting direction, such as the ground, the compensation coefficient is correspondingly reduced to avoid over-enhanced false edges.

[0089] For example, when extracting edge features, consider a typical scenario: a robot encounters a white wall partially exposed to direct sunlight at a corridor corner. The traditional Canny operator might generate numerous false edges at the light-shadow boundary. In this approach, the improved directional gradient operator M(x,y) automatically adjusts its response intensity based on the lighting projection vector: for physical structural edges such as corners (usually perpendicular to the ground), the directional enhancement coefficient γ highlights these features due to the large angle between their gradient direction and the lighting direction; while for light-shadow boundaries (usually parallel to the lighting direction), their response values ​​are appropriately suppressed, thus preventing these unstable features from interfering with subsequent localization.

[0090] For example, in the stereo matching stage of feature points, consider a scene containing a glass window: due to reflection and perspective effects, the images captured by the left and right cameras may have local brightness differences. The illumination-adaptive matching weight φ(x,y) introduced in this scheme can dynamically adjust the matching strategy: in reflective areas, since the gradient direction is highly correlated with the illumination direction, the matching weight will automatically decrease, and the algorithm will rely more on the surrounding structural features for matching; while in stable structural areas such as window frames, since the gradient direction has a low correlation with the illumination direction, the matching weight will be increased accordingly to ensure accurate matching of these key feature points.

[0091] Regardless, the core objective of this solution is to extract stable and reliable visual features in scenarios with drastic lighting changes, providing a reliable perceptual foundation for the precise positioning of mobile robots. As the examples above demonstrate, each algorithm module in this solution is meticulously designed to effectively handle various complex lighting conditions encountered in real-world applications.

[0092] The feature unit extraction module sequentially selects points to be analyzed along the edge contour. For each point to be analyzed, it selects 25 pixels forward and 25 pixels backward to form a local curve segment. The coordinate sequence of contour points within the local curve segment is converted into a parametric equation, the parameter values ​​of which are determined by the cumulative arc length of the contour points. Cubic spline interpolation is used to continuously reconstruct the local curve segment to obtain the local curve equation. Based on the local curve equation, the tangent vector at the point to be analyzed, as well as the tangent vectors at 10 pixels forward and 10 pixels backward, are calculated. The angles between the tangent vector of the point to be analyzed and the tangent vectors of the forward and backward points are calculated. When the angle between the tangent vector of the point to be analyzed and the tangent vector of the forward point increases and the angle between the tangent vector and the tangent vector of the backward point decreases, the point to be analyzed is marked as a potential inflection point.

[0093] For the potential inflection point, calculate its local curvature value; when the local curvature value shows a trend of changing from increasing to decreasing or from decreasing to increasing, calculate the curvature acceleration at the potential inflection point; select the potential inflection point where the curvature acceleration is the local maximum value as the inflection point; for the inflection point, calculate the angle between the depth gradient direction and the contour tangent vector at its location; when the angle between the depth gradient direction and the contour tangent vector is in the range of 75 degrees to 105 degrees, upgrade the confidence of the inflection point to a high-confidence inflection point. Analyze the curve segment characteristics between the high-confidence inflection points; when the curvature change value of the curve segment between two adjacent high-confidence inflection points is greater than twice the average curvature of the curve segment, retain these two high-confidence inflection points; when the curvature change value of the curve segment is less than twice the average curvature of the curve segment, calculate the depth difference between these two high-confidence inflection points; if the depth difference is greater than 30% of the average depth value of the curve segment, retain these two high-confidence inflection points, otherwise retain the high-confidence inflection point with the larger curvature acceleration.

[0094] Furthermore, the contour feature construction module uses the edge contour segment between two adjacent inflection points as feature units, and calculates the geometric shape parameters and relative depth difference of the feature units, as follows:

[0095] Obtain the sampling point sequence of the feature unit, and calculate the arc length between adjacent sampling points; accumulate the arc lengths between adjacent sampling points to obtain the total arc length L of the feature unit; perform principal component analysis based on the spatial coordinates of the sampling points to obtain the principal direction vector V of the feature unit; calculate the set of vertical projection distances D from the sampling points to the principal direction vector V; calculate the maximum value Dmax in the set of vertical projection distances D; calculate the ratio R of the maximum value Dmax to the total arc length L.

[0096] The sampling point sequence is fitted with a least-squares curve to obtain the curvature change function K(s) of the fitted curve; the mean Km and variance Kv of the curvature change function K(s) are calculated as the curvature feature parameters of the feature unit.

[0097] Calculate the direction vector U of the line connecting the start point and the end point of the feature unit; perform a cross product between the direction vector U and the main direction vector V to obtain the normal vector N of the feature unit; calculate the depth gradient values ​​G1 and G2 on both sides of the feature unit along the positive and negative directions of the normal vector N.

[0098] Based on the sampling point sequence of the feature unit, the depth difference of each sampling point relative to the starting point is calculated; according to the normalized position s of the sampling point in the arc length direction, the depth difference function H(s) is calculated; the first derivative H'(s) and the second derivative H'(s) of the depth difference function H(s) are calculated; when the absolute value of the first derivative H'(s) of the depth difference function H(s) at a certain point s0 is greater than 3 times the average rate of change of the depth difference function H(s), the sampling point density is increased at position s0; the number of newly added sampling points is proportional to the absolute value of the first derivative H'(s); based on the three-dimensional coordinates of the sampling points, the spatial curvature function C(s) of the feature unit is calculated; the spatial curvature function C(s) and the depth difference function H(s) are correlated to obtain the correlation function A(s);

[0099] Construct a feature vector F, which includes the following components: total arc length L, mean curvature Km, curvature variance Kv, depth gradient values ​​G1 and G2, mean first derivative of depth difference function Hm', mean second derivative of depth difference function Hm”, and mean of correlation function Am.

[0100] Furthermore, normalizing the geometric shape parameters and depth feature parameters yields a lighting-independent contour feature descriptor.

[0101] For example, during indoor navigation of a mobile robot, when the robot is in an office environment, its binocular cameras capture a continuous edge profile in a corner area. This edge profile includes a column structure protruding from the wall, and the junction of the column and the wall forms a feature unit. First, a series of sampling points are acquired on this feature unit. Due to the significant depth change in the transition area between the column and the wall, the sampling point density is increased in this area. The distances between adjacent sampling points are calculated and summed to obtain the total arc length of the feature unit.

[0102] Next, the geometric features of the feature unit are analyzed. Principal component analysis is used to determine the principal direction of the feature unit, i.e., the direction in which the column extends. The vertical projection distance from the sampling point to this principal direction is calculated; these distances reflect the degree of column protrusion. Curve fitting is performed on the sampling point sequence to obtain a curvature variation function, which describes the curvature variation characteristics at the transition between the column and the wall. Simultaneously, the depth variation characteristics of the feature unit are analyzed. The normal vector direction of the feature unit is calculated; this direction is perpendicular to the extension direction of the column. Examining the depth variation along the normal vector direction reveals a significant depth gradient in the transition region between the column and the wall. By calculating the depth difference between the sampling point and the starting point, a function describing the depth variation is obtained.

[0103] Finally, the total arc length, curvature features, and depth gradient features of the feature units are combined into a feature vector. This feature vector describes both the geometric shape of the column structure and its three-dimensional spatial features, which can be used for subsequent scene recognition and localization. The advantage of this method is that even when lighting conditions change, the extracted features remain stable because the depth features are unaffected by lighting.

[0104] It should be noted that when the absolute value of the first derivative H'(s) of the depth difference function H(s) at a certain point s0 is greater than three times the average rate of change of the depth difference function H(s), a sampling point density increase strategy is implemented, specifically including:

[0105] Centered on position s0, take the interval [s0-δ, s0+δ] with an arc length of 5% of the total arc length in front and behind as the local encryption interval;

[0106] Calculate the maximum value of the first derivative H'max of the depth difference function H(s) within the local encryption interval; calculate the increase in sampling points n based on the maximum value of the first derivative H'max.

[0107] When H'max is located in the first 10th percentile of the first derivative of the depth difference function H(s), the number of sampling points increased by n is 8.

[0108] When H'max is located in the first 5% quantile of the first derivative of the depth difference function H(s), the number of sampling points increased by n is 12.

[0109] When H'max is located in the first 1% quantile of the first derivative of the depth difference function H(s), the number of sampling points increased by n is 16.

[0110] Within the local encryption interval [s0-δ, s0+δ], the positions of n newly added sampling points are determined based on the distribution of the first derivative of the depth difference function H(s):

[0111] First, the local encrypted interval is divided into n+1 sub-intervals;

[0112] Then, within each subinterval, the specific location of the new sampling point is determined based on the location of the local maximum value of the first derivative H'(s);

[0113] When the change in the first derivative H'(s) within the sub-interval is less than 10% of H'max, the sampling point is set at the midpoint of the sub-interval.

[0114] For the newly added sampling point, its three-dimensional spatial coordinates and depth value are obtained: using the spatial coordinates of the original sampling points, cubic spline interpolation is used to calculate the spatial coordinates of the newly added sampling point; based on the depth values ​​of the adjacent original sampling points, linear interpolation is used to calculate the depth value of the newly added sampling point.

[0115] Add the newly added sampling points to the sampling point sequence and update the sampling point set of the feature unit:

[0116] The new sampling points are inserted into the original sampling point sequence according to the arc length parameter order;

[0117] Recalculate the depth difference function H(s) that includes the newly added sampling points;

[0118] Verify the accuracy of the updated depth difference function H(s) in the local encrypted region. When the reconstruction error of the depth difference is greater than the preset threshold, return the number of sampling points to be increased.

[0119] When performing scene matching, the global localization module first extracts the geometric shape parameter vector and depth feature vector of each contour feature descriptor in the current frame. For each contour feature to be matched, a hierarchical matching strategy is executed within a preset search range. When the depth difference sequence of features shows strong local consistency, the system prioritizes the depth feature vector for initial screening, determining initial matching candidates by calculating the correlation coefficient of the depth sequence. If the depth features are not significant enough, the geometric shape parameters are used for matching first. In this case, the system calculates a weighted combination score of the normalized arc length ratio, curvature distribution similarity, and projection length ratio. When the depth feature correlation coefficient of a candidate matching pair exceeds 0.85 and the combination score of the geometric shape parameters is higher than a set threshold, it is marked as a reliable matching pair. If only one of the conditions is met, it is marked as a matching pair to be verified. The system further checks the temporal consistency of the contour segment in adjacent frames. Only when similar feature change trends can be observed in three or more consecutive frames is it confirmed as a valid match. For contour segments whose geometry changes significantly but whose depth features remain stable, the system examines their local environment structure. If the changes conform to the geometric projection rules caused by the change in viewpoint, the matching pair is retained. Finally, spatial consistency verification is performed on all confirmed matching pairs, requiring that the relative positional relationship between adjacent matching pairs should remain stable. When a matching pair is found to violate this constraint, the system removes it from the matching results, thereby ensuring the reliability of the final matching results.

[0120] To determine the global position of the mobile robot, a dynamically updated spatiotemporal feature graph structure is first constructed. Nodes in the graph represent contour features observed at different times and their corresponding robot pose information, while edges between nodes represent feature matching relationships and pose constraints. When a new observation frame arrives, the system adds the contour features obtained in that frame and the current coarse pose as new nodes to the graph structure and establishes association edges with historical features. If a feature is observed in multiple consecutive frames, temporal constraint edges are established for the poses of adjacent frames.

[0121] The system employs a block matrix structure to store pose constraint information, where diagonal blocks represent the uncertainty of pose in a single frame, and off-diagonal blocks represent the correlation of pose between frames. For each pair of matched features, their relative transformation relationship in 3D space is calculated to construct the constraint equations for pose graph optimization. The constraint equations include feature matching terms and pose continuity terms, where the feature matching term reflects the geometric consistency of contour features across different observation frames, and the pose continuity term ensures smooth changes in pose between adjacent time steps.

[0122] During the optimization process, the system uses a weighted least squares method to iteratively solve for the optimal pose. The weights of the feature matching constraints are dynamically adjusted based on the geometric similarity and depth consistency of the matching pairs, while the weights of the pose continuity constraints are determined based on the motion state and feature distribution. The system updates the constraint weights in each iteration and uses sparse matrix factorization to efficiently solve the optimization problem.

[0123] The system employs a sliding window strategy to maintain the optimization window, retaining data from the most recent 20 frames by default. After optimization convergence, the pose estimation results within the window are fused with the previously saved pose trajectories to obtain the global position.

[0124] For feature matching constraints, the system calculates the relative transformation relationship of each pair of matched contour features in 3D space, converting it into a constraint equation for the robot's pose. When two contour features are observed to be successfully matched, the system calculates the theoretically required relative pose transformation of the robot based on their 3D positions and orientations in their respective coordinate systems, and adds this constraint to the optimization problem. Next, for pose continuity constraints, the system uses the robot's kinematic model to calculate motion constraints based on pose differences between adjacent time points. When the robot uses differential drive, the system converts wheel speedometer data into pose increment predictions and constructs constraint conditions with the actual pose differences. During optimization, the system assigns dynamic weights to each constraint: for feature matching constraints, higher weights are assigned when the geometric and depth features of the matched pair are more similar; for pose continuity constraints, larger weights are assigned when the robot's motion is smooth and its speed is low, while the weights are appropriately reduced when the robot is rapidly turning or accelerating / decelerating. The system employs the Iterative Closest Point (ICP) approach. In each iteration, it first re-evaluates the accuracy of feature matching based on the current pose estimation, eliminating unreasonable matching pairs. Then, it solves a least-squares problem by simultaneously applying all valid constraints to solve for the pose increment. When the pose increment is less than a preset threshold in a given iteration, the optimization is considered converged, and the final position estimation result is output.

[0125] For example, taking a specific shopping mall scenario, the robot needs to provide navigation services between the lobby and various shops on the first floor. When the robot enters the bright atrium area from the darker entrance area, the system performs the following positioning process:

[0126] In the entrance area, the main contour features collected by the robot include the door frame structure of the automatic glass door (vertical contour), the horizontal contour where the ceiling and floor meet, and the edge contours of the billboards on both side walls. As the robot moves into the atrium through a gradually changing lighting area, the system prioritizes tracking these structural contours with significant depth differences. In particular, the junction of the vertical contour of the automatic door frame and the floor forms a stable inflection point feature that can still be reliably detected during changes in lighting. After the robot enters the atrium area, new contour features are added, including the atrium columns, the boundary contours of the shop facades, and the edges of the light and shadow projected by the atrium skylight. At this point, the system establishes two sets of key nodes in the graph structure: one set representing the features and corresponding poses of the entrance area, and the other set representing the new features and current poses of the atrium area. By tracking the fixed structural contours extending from the entrance area to the atrium (such as the wall junctions and the edges of the floor textures), the system establishes reliable associated edges between these two sets of nodes.

[0127] When the robot is near a pillar in the atrium, the system simultaneously observes three key contour features: the pillar's edge contour (vertical linear feature), the dividing lines of the floor tiles (horizontal grid feature), and the outline of the skylight frame overhead (fixed height feature). These three sets of features form a stable three-dimensional reference structure in space. If the robot detects a deviation in its pose at this location (e.g., the odometer shows a 0.5-meter shift to the left), the system corrects itself using the relative positional relationships of these three sets of features: first, it verifies whether the projected position of the pillar's edge aligns with the intersection of the floor tile grid; then, it checks whether the projection of the skylight frame from the current viewpoint matches the position of the top of the pillar.

[0128] If a systematic deviation is found between the actual observed feature locations and the expected locations, such as all features shifting to the right by approximately 0.4 meters, it indicates that the odometry has accumulated errors. In this case, the graph optimization algorithm adjusts the robot's pose estimation to best match the observed feature locations with known reference structures in the environment. Through this multi-constraint optimization, the system ultimately corrects the robot's position to a 0.4-meter rightward shift, which aligns with the reference feature locations in the actual environment. This correction process is continuous and smooth, ensuring the robot maintains accurate positional awareness at all times, enabling stable operation even in transitional areas with drastically changing lighting conditions.

[0129] In summary, the binocular vision global localization system for mobile robots based on embodiments of the present invention has been clarified. By performing adaptive exposure compensation on the binocular images from the left and right cameras, it extracts illumination-independent edge contour feature points and generates a high-quality depth feature map by combining this with parallax calculation. Simultaneously, feature units are constructed based on inflection points, thereby determining the global position of the mobile robot. This approach can, to some extent, solve the problem of unstable visual feature extraction and decreased global localization accuracy caused by drastic changes in illumination intensity in multi-area switching scenarios.

[0130] like Figure 2 As shown, a binocular vision global localization method for mobile robots is provided, which includes:

[0131] The system acquires binocular images from the left and right cameras of the mobile robot and obtains the pose information and illumination sensor data of the mobile robot, wherein the pose information includes the three-dimensional spatial coordinates and attitude angle data of the mobile robot.

[0132] Adaptive exposure compensation is performed on the binocular image based on the illumination sensor data, edge contour feature points in the image are extracted, and depth information of the edge contour feature points is calculated based on parallax to generate an illumination-compensated depth feature map.

[0133] In the depth feature map, the inflection point position of the object edge contour is identified, the edge contour segment between two adjacent inflection points is taken as the feature unit, the geometric shape parameters and relative depth difference of the feature unit are calculated, and a light-independent contour feature descriptor is constructed.

[0134] The contour feature descriptor is used for scene matching, and combined with the pose information, a graph structure optimization algorithm is used to determine the global position of the mobile robot.

[0135] Here, those skilled in the art will understand that the specific operations of each step in the above-described binocular vision global localization method for mobile robots have been referenced above. Figure 1 The description of the binocular vision global localization system for mobile robots is detailed here, and therefore, its repeated description will be omitted.

[0136] In summary, the binocular vision global localization method for mobile robots based on embodiments of the present invention has been clarified. By performing adaptive exposure compensation on the binocular images from the left and right cameras, light-independent edge contour feature points are extracted, and a high-quality depth feature map is generated by combining this with parallax calculation. Simultaneously, feature units are constructed based on inflection points, thereby determining the global position of the mobile robot. This approach can, to some extent, solve the problem of unstable visual feature extraction and decreased global localization accuracy caused by drastic changes in light intensity in multi-region switching scenarios.

Claims

1. A binocular vision global positioning system for a mobile robot, characterized in that, include: The data acquisition module is used to acquire binocular images from the left and right cameras of the mobile robot, and to obtain the pose information and illumination sensor data of the mobile robot, wherein the pose information includes the three-dimensional spatial coordinates and attitude angle data of the mobile robot. An adaptive exposure compensation module is used to perform adaptive exposure compensation on the binocular image based on the light sensor data, extract edge contour feature points in the image, calculate the depth information of the edge contour feature points based on parallax, and generate a depth feature map after light compensation. The feature unit extraction module is used to identify the inflection point position of the object edge contour in the depth feature map and take the edge contour segment between two adjacent inflection points as feature units. The contour feature construction module is used to calculate the geometric shape parameters and relative depth difference of the feature unit and construct a lighting-independent contour feature descriptor. The global localization module is used to perform scene matching using the contour feature descriptor, and combined with the pose information, to determine the global position of the mobile robot using a graph structure optimization algorithm. The adaptive exposure compensation module performs exposure compensation on the binocular image, including a lighting environment analysis and projection calculation stage, an adaptive exposure compensation stage, a directional edge feature extraction stage, and a stereo matching and depth calculation stage. The adaptive exposure compensation stage applies to each pixel of the binocular image. By combining its local gradient information and illumination projection information, the adaptive exposure compensation coefficient is calculated. ; When the local gradient direction is perpendicular to the illumination direction, the compensation coefficient reaches its maximum value, and the compensated image pixel value is calculated. The adaptive exposure compensation coefficient The calculation is shown in the following formula: in, This is the compensation coefficient at pixel (x, y), with a value range of [0, 1]. Let be the gradient vector at pixel (x, y). Let be the projection vector of the illumination direction onto the image plane. Let be the magnitude of the gradient vector. Let be the magnitude of the light projection vector. For adaptive gain parameters; The directional edge feature extraction stage extracts edges from the compensated image and calculates the gradient operator. Calculate the Harris response value on the detected edge contour. To locate feature points; Based on the calculated gradient operator value Harris response value Construct a feature point reliability metric Select reliability metrics Greater than the adaptive threshold The points are used as candidate feature points; Let the set of feature points obtained from the selection be denoted as . , used for the stereo matching process in the stereo matching and depth calculation stage.

2. The binocular vision global positioning system for mobile robots according to claim 1, characterized in that, The lighting environment analysis and projection calculation stage calculates the scene's main lighting direction vector using a weighted vector synthesis method. and overall light intensity And the calculated main illumination direction vector The image plane is transformed using the camera projection matrix to obtain a two-dimensional illumination projection vector. .

3. The binocular vision global positioning system for mobile robots according to claim 1, characterized in that, The feature unit extraction module identifies inflection points including: The points to be analyzed are selected along the edge contour of the depth feature map, and local curve segments are formed by the forward and backward pixels of each point to be analyzed. The coordinates of the contour points of the local curve segment are converted into parametric equations, and the local curve equations are reconstructed through cubic spline interpolation. The tangent vectors of the point to be analyzed and its forward and backward pixels are calculated based on the local curve equation, and potential inflection points are marked by the change in the angle between the tangent vectors. Calculate the local curvature value of potential inflection points. When the local curvature value shows a changing trend, calculate the curvature acceleration and select the potential inflection point with the largest curvature acceleration as the inflection point. Calculate the angle between the depth gradient direction and the contour tangent vector at the inflection point. When the angle is within a preset range, mark the inflection point as a high-confidence inflection point, and optimize the retention of high-confidence inflection points based on the curvature change value and depth difference.

4. The binocular vision global positioning system for mobile robots according to claim 1, characterized in that, The sampling point sequence based on the feature unit is used to calculate feature parameters and then normalize to obtain the illumination-independent contour feature descriptor. The feature parameters include the arc length, principal direction, curvature feature, depth gradient of the sampling point, and the correlation function of the sampling point sequence.

5. The binocular vision global positioning system for mobile robots according to claim 1, characterized in that, The global positioning module determines the global position of the mobile robot by including: A spatiotemporal feature map is constructed, along with two types of core constraints; these two types of core constraints include feature matching constraints and pose continuity constraints. The two types of core constraints are transformed into optimization problems. Through iterative optimization, the pose estimation at each time step is adjusted to ensure that the overall constraints are best satisfied. The optimized pose sequence is fused with the previously saved trajectory to obtain the robot's position in the global coordinate system.

6. The binocular vision global positioning system for mobile robots according to claim 5, characterized in that, The feature matching constraint is used to describe the spatial relationship of the same contour features at different times, and the pose continuity constraint ensures the smoothness of the robot's motion. By performing weighted iterative optimization on these two types of constraints until the pose estimation converges, an accurate position result is obtained.

7. A binocular vision global localization method for a mobile robot, based on the binocular vision global localization system for mobile robots as described in claims 1-6, characterized in that, include: The system acquires binocular images from the left and right cameras of the mobile robot and obtains the pose information and illumination sensor data of the mobile robot, wherein the pose information includes the three-dimensional spatial coordinates and attitude angle data of the mobile robot. Adaptive exposure compensation is performed on the binocular image based on the illumination sensor data, edge contour feature points in the image are extracted, and depth information of the edge contour feature points is calculated based on parallax to generate an illumination-compensated depth feature map. In the depth feature map, the inflection point position of the object edge contour is identified, the edge contour segment between two adjacent inflection points is taken as the feature unit, the geometric shape parameters and relative depth difference of the feature unit are calculated, and a light-independent contour feature descriptor is constructed. The contour feature descriptor is used for scene matching, and combined with the pose information, a graph structure optimization algorithm is used to determine the global position of the mobile robot.

Images