Robot obstacle avoidance control system and method based on visual identification

Abstract

The invention provides a robot obstacle avoidance control system and method based on visual identification, and relates to the technical field of image identification control, and the method comprises the steps: collecting image data in front of a robot, constructing a spatial-temporal feature point sequence, calculating obstacle index data, dividing obstacles into obstacle types, respectively constructing prediction models, and meanwhile, carrying out the obstacle avoidance of the robot. The method comprises the following steps: grouping obstacles based on obstacle indexes, constructing a spatial association graph, carrying out hierarchical estimation on obstacle states, finally, constructing an obstacle avoidance priority decision tree based on a prediction model and a state estimation result, and controlling the motion direction and speed of a robot according to the output of the decision tree. Therefore, the problems that the obstacle state in a complex shielding scene is difficult to accurately predict and the obstacle avoidance strategy is not flexible can be solved, and the obstacle avoidance precision and safety of the robot in a complex environment are improved.

Description

Technical Field

[0001] This invention relates to the field of image recognition control technology, and more specifically, to a robot obstacle avoidance control system and method based on vision recognition. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence and robotics, computer vision-based obstacle avoidance systems have gradually become an important research direction in the field of intelligent robotics. Robot obstacle avoidance control technology is widely used in industrial automation, autonomous driving, and service robots. Its core is to perceive environmental information through sensors and combine this with motion planning algorithms to effectively avoid dynamic or static obstacles. In visual recognition technology, the combination of cameras and depth sensors provides robots with rich environmental perception capabilities. In particular, the emergence of RGB-D sensors enables robots to simultaneously acquire color and depth information from images, providing an important foundation for multi-scale feature extraction, obstacle detection, and trajectory prediction. Currently, visual recognition-based obstacle avoidance control methods typically rely on the collaborative work of sub-modules such as feature extraction and tracking, motion trajectory prediction, obstacle state estimation, and obstacle avoidance strategy optimization. However, with the increasing complexity of environments, such as multiple obstacle occlusions, complex movements of dynamic obstacles, and discontinuous depth information, traditional obstacle avoidance methods face severe challenges in terms of accuracy, real-time performance, and stability.

[0003] In existing technologies, most robot obstacle avoidance methods rely on feature extraction and analysis based on a single scale or a single information source (such as RGB images or depth maps alone). These methods often have shortcomings when dealing with complex dynamic environments: First, in terms of multi-scale feature extraction, due to the lack of comprehensive analysis of multi-scale pyramid image features, existing methods struggle to capture detailed obstacle information at different scales, especially in dynamically occluded regions, where feature extraction exhibits poor robustness. Second, for the analysis of obstacle occlusion states and levels, existing technologies mostly rely on static occlusion models, which are ill-suited to handling dynamic changes in occlusion and complex scenarios with multi-level occlusion. Furthermore, in terms of trajectory prediction for dynamic obstacles, existing technologies typically employ a single model (such as a simple Kalman filter or particle filter) for trajectory estimation, which shows significant limitations when dealing with obstacles undergoing unstable or turning motions. Especially in cases of severe occlusion or complex motion states, the reliability of existing methods for obstacle motion prediction and obstacle avoidance decisions decreases significantly, leading to inefficient robot path planning and even the risk of obstacle avoidance failure. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention is proposed. This invention provides a vision-based robot obstacle avoidance control system and method, which can, to some extent, solve the problem of difficulty in effectively handling multi-level occlusion and dynamic obstacle movement in complex environments due to deficiencies in multi-scale feature extraction, dynamic occlusion processing, and trajectory prediction methods.

[0005] According to one aspect of the present invention, a vision-based obstacle avoidance control method for robots is provided, comprising:

[0006] A multi-scale edge feature map of the image in front of the robot is acquired, a spatiotemporal feature point sequence is constructed on the multi-scale edge feature map, and obstacle index data is calculated.

[0007] Based on the obstacle index data, obstacles are divided into stable moving obstacles and unstable moving obstacles. A motion trajectory prediction model is established for the stable moving obstacles, and a multi-hypothesis state estimation model is established for the unstable moving obstacles.

[0008] Based on the obstacle index data, obstacles are grouped, a spatial relationship map between obstacles is established, and the obstacle state is estimated hierarchically based on the spatial relationship map. Combined with the prediction model, the trajectory of obstacles at different occlusion levels is tracked.

[0009] Based on the motion trajectory prediction model, the multi-hypothesis state estimation model, and the estimation results of the obstacle state, an obstacle avoidance priority decision tree is constructed, and the robot's motion direction and speed are controlled according to the output of the decision tree.

[0010] Furthermore, based on the stable moving obstacle, a motion trajectory prediction model based on Kalman filtering is adopted, using position, velocity, and acceleration as state variables, using observation data from the past time period to predict the trajectory, and dynamically adjusting the noise covariance matrix according to the reliability of feature point tracking and feature matching.

[0011] Furthermore, based on the aforementioned multi-hypothesis state estimation model, through the assumptions of three basic motion modes—uniform motion, uniformly accelerated motion, and turning motion—each mode corresponds to an independent Kalman filter, and the probability weights of each mode are dynamically adjusted using an interactive multi-model algorithm.

[0012] Furthermore, the obstacle index data includes motion continuity index, occlusion status index, and occlusion depth index.

[0013] Furthermore, obstacles are grouped based on the occlusion status index to determine whether occlusion relationships exist, and obstacles with occlusion relationships are divided into occlusion groups. Within each occlusion group, a spatial association graph is constructed to record the occlusion attributes between obstacles, and abnormal relationships are eliminated through a temporal consistency check.

[0014] Furthermore, based on the occlusion depth index, the analysis of multiple occlusion situations is conducted. When the depth jump in the area is significant and the feature points exhibit a multi-layered disappearance-reappearance pattern, it is determined to be a multiple occlusion area and the processing mode is activated. By establishing a directed occlusion relationship graph and combining the occlusion area ratio, depth jump amplitude, and occlusion duration, the occlusion level of the obstacle is determined.

[0015] Furthermore, a differentiated tracking strategy is adopted based on different occlusion levels: the foreground layer uses full feature point tracking, the intermediate layer uses a prediction-validation framework to handle feature point disappearance and position correction, and the background layer relies on motion model prediction and uses historical trajectory matching for feature points.

[0016] Furthermore, the tracking results of the differentiated tracking strategy are input into the extended Kalman filter framework, and combined with inter-layer constraints, the obstacle state is optimized and estimated to reconstruct the complete scene structure and restore the geometric contour of the occluded area.

[0017] Furthermore, an obstacle avoidance priority decision tree is constructed based on the motion trajectory prediction model, the multi-hypothesis state estimation model, and the estimation results of the obstacle state;

[0018] The obstacle avoidance priority decision tree divides the surrounding environment into sectors, calculates risk scores based on the obstacle's movement type, occlusion status, and confidence level, dynamically adjusts sector weights, and selects the direction with the lowest weight for obstacle avoidance.

[0019] According to another aspect of the present invention, a vision-based obstacle avoidance control system for robots is provided, comprising:

[0020] The acquisition module collects image data in front of the robot, constructs a spatiotemporal feature point sequence, and calculates obstacle index data;

[0021] The prediction module classifies obstacles into obstacle types based on the obstacle index data and constructs prediction models for each type.

[0022] The classification module groups obstacles based on the obstacle index data, establishes a spatial relationship map between obstacles, performs hierarchical estimation of obstacle status, and combines a prediction model to track the trajectory of obstacles at different occlusion levels.

[0023] The control module constructs an obstacle avoidance priority decision tree based on the prediction model and the estimation results of the obstacle state, and controls the robot's movement direction and speed according to the output of the decision tree.

[0024] Compared with existing technologies, this invention significantly improves the robot's obstacle detection, occlusion handling, and obstacle avoidance decision-making capabilities in complex dynamic environments by combining a multi-hypothesis state estimation model with an obstacle avoidance priority decision tree. It can determine the motion characteristics and spatial relationships of obstacles at different scales and occlusion levels, and dynamically adjust obstacle avoidance strategies, enabling the robot to achieve efficient and stable path planning and navigation even in high-risk, multi-occluded, and unstable motion scenarios. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0026] Figure 1 This is a flowchart of a robot obstacle avoidance control method based on vision recognition according to an embodiment of the present invention;

[0027] Figure 2 This is a flowchart illustrating the prediction of motion trajectories according to an embodiment of the present invention. Detailed Implementation

[0028] Hereinafter, exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments of the present invention. It should be understood that the present invention is not limited to the exemplary embodiments described herein.

[0029] As mentioned in the background section, existing technologies have two main problems: First, the occlusion relationship of multiple obstacles in complex environments makes it difficult to accurately predict the motion state of obstacles, especially when there are multiple occlusions. Traditional methods are prone to the accumulation of prediction errors, thereby reducing obstacle avoidance accuracy. Second, the state estimation and obstacle avoidance strategies for obstacles at different occlusion levels lack specificity and cannot dynamically adjust weight allocation and decision logic, resulting in an unsafe and inefficient obstacle avoidance process.

[0030] Figure 1 This is a flowchart of a vision-based obstacle avoidance control method for robots according to an embodiment of the present invention. Figure 1 As shown, the robot obstacle avoidance control method based on vision recognition includes:

[0031] S1: Collect multi-scale edge feature maps of the image in front of the robot, construct a spatiotemporal feature point sequence on the multi-scale edge feature maps, and calculate obstacle index data.

[0032] A depth camera system mounted on the front of the robot is used to simultaneously acquire RGB and depth images. The depth camera system has a field of view of 120 degrees, a resolution of 1920×1080 pixels, and a sampling frequency of 60 Hz.

[0033] The depth camera uses a binocular structured light scheme, with two infrared cameras spaced 8 centimeters apart. The structured light projector is located between the two cameras, enabling the acquisition of depth data with an accuracy better than 1 centimeter within a range of 0.5 meters to 8 meters.

[0034] After acquiring the RGB image, a conventional multi-scale feature extraction method is used to process it. First, a feature pyramid with four scales is constructed, with a scaling factor of 1.6 between adjacent scale layers. Images at different scales are generated layer by layer through Gaussian filtering and downsampling. The resolution of the original input image is denoted as layer zero; the size of the first layer is 1 / 1.6 of the original image; the size of the second layer is 1 / 1.6 of the first layer; and the size of the third layer is 1 / 1.6 of the second layer. This progressive downsampling ensures that feature information is captured at different scales.

[0035] At each scale level, both Histogram of Oriented Gradients (HOG) features and Local Binary Pattern (LBP) features are extracted simultaneously. For HOG feature extraction, the horizontal and vertical gradients of the image are first calculated. Then, the gradient direction is divided into nine equal directional intervals within a 360-degree range. The gradient magnitude of each interval is calculated using bilinear interpolation, forming a 9-dimensional HOG descriptor. For LBP feature extraction, a circular neighborhood with a radius of 2 is selected, and eight neighboring points are sampled. Binary encoding is generated by comparing the grayscale values ​​of the center pixel with those of its neighbors, resulting in a 256-dimensional feature vector. This vector is then converted to an equivalent pattern using a mapping table, reducing the feature dimension to 59.

[0036] For each scale layer image, a 9×9 pixel sliding window is used for feature point extraction, with a sliding step size of 4 pixels between windows to balance computational efficiency and feature coverage density. At each sliding window position, the extracted HOG features and LBP features are concatenated to form a joint feature vector. When the norm of this feature vector is greater than a preset threshold, the center position of the window is marked as a candidate feature point.

[0037] To establish the correspondence between feature points at different scales, a spatial location mapping method is employed. Specifically, for a feature point in scale s, its projected position in the adjacent scale s+1 is calculated based on the scale factor. If a feature point exists within a 3×3 pixel neighborhood of the projected position, a scale association is established between these two feature points. In this way, feature points from all scales are linked together, ultimately forming a multi-scale feature point set.

[0038] A multi-scale feature point set was tracked on a sequence of 10 consecutive frames of images. An improved KLT optical flow algorithm was used to calculate the motion vector of the feature points, with a time window length of 0.5 seconds.

[0039] Specifically, the improved KLT optical flow algorithm tracks a multi-scale feature point set based on a sequence of 10 consecutive image frames, with an inter-frame time interval of 0.05 seconds and a total time window length of 0.5 seconds. When processing feature point tracking between any two adjacent frames, the following steps are performed:

[0040] First, motion prediction is performed. For each feature point, its motion state in the first three frames is analyzed, and its approximate position in the next frame is predicted by calculating its velocity and acceleration. When the feature point has a large motion velocity, the predicted position will be used as the center of the subsequent search window, thereby improving the success rate of large displacement tracking.

[0041] An image pyramid was then constructed using a three-layer structure, with the sampling ratio between adjacent layers dynamically adjusted based on the image content. When the image region contained rich texture, the sampling ratio was set to 0.6 to preserve more detail; when the region was relatively smooth, the sampling ratio was reduced to 0.4 to improve computational efficiency. During the construction of each pyramid layer, image patches underwent normalization preprocessing, reducing the impact of illumination variations by subtracting the local mean and dividing by the standard deviation.

[0042] Feature points are tracked layer by layer starting from the top of the pyramid. In each layer, the size and shape of the search window are first determined. The window size is dynamically adjusted according to the movement speed of the feature points, with a basic size of 15×15 pixels, which can be expanded to 21×21 pixels when fast movement is detected. If the feature points exhibit obvious directional movement, the window is adjusted to an ellipse along the direction of movement to improve search efficiency.

[0043] Within a defined search window, the displacement of feature points is calculated by solving the optical flow equations. Specifically, the spatial gradient (horizontal and vertical) and temporal gradient of each pixel within the window are first calculated, and then a 2×2 normal equation system is constructed. This equation system is solved iteratively to obtain the displacement increment of the feature points. The iteration of the current layer ends when the displacement increment is less than 0.1 pixels, or when the maximum number of iterations (set to 20) is reached.

[0044] After tracking one layer of the pyramid, the displacement results are projected onto the next layer as the initial position. During the projection process, the motion information of the feature points is simultaneously transmitted to guide the window setting and iterative optimization of the next layer.

[0045] To address occlusion issues, a reliability assessment is performed after each tracking iteration. First, the normalized correlation coefficient of the image patch before and after tracking is calculated, with a base threshold of 0.8. Simultaneously, the relative positional relationship between feature points and their neighboring reference points (other feature points within 20 pixels) is analyzed. If the correlation coefficient falls below the threshold or the relative positional relationship changes significantly, the feature point is marked as potentially occluded. For marked feature points, their last reliable position is recorded, and they are continuously monitored in subsequent frames until the occlusion is detected to have disappeared, at which point normal tracking resumes.

[0046] For the feature points that are ultimately retained, their displacement vectors between adjacent frames are recorded as the basis for calculating motion metrics.

[0047] The following metrics are calculated based on the motion vectors of feature points:

[0048] Motion continuity index: The stability of motion is characterized by calculating the variance and first derivative of the motion vector of feature points within a time window. The index value ranges from [0,1].

[0049] Occlusion status index: Based on the disappearance-appearance pattern of feature points and the continuity analysis of the depth map, it determines whether the feature points are in an occlusion state. The index is a binary quantity.

[0050] Occlusion depth index: When occlusion is detected, the discontinuous boundaries in the depth map are extracted, and the depth jump threshold is set to 10 cm; the gradient intensity of the boundary region is calculated using the Sobel operator with a window size of 5×5 pixels; the gradient intensity is mapped to the [0,1] interval and weighted and fused with the depth jump variable with a weight ratio of 0.4:0.6; the cumulative depth change in the occluded region is statistically analyzed to obtain the final occlusion depth index.

[0051] S2: Based on the obstacle index data, the obstacles are divided into stable moving obstacles and unstable moving obstacles. A motion trajectory prediction model is established for the stable moving obstacles, and a multi-hypothesis state estimation model is established for the unstable moving obstacles.

[0052] Obstacles are initially classified based on the motion continuity index of feature points. The motion of each feature point is tracked within a 0.5-second time window (10 frames). First, the position of the feature point in each frame is recorded, and its velocity and acceleration between adjacent frames are calculated. The changes in these motion parameters are analyzed. When the velocity change of a feature point remains within 20% and the acceleration change remains within 30% over five consecutive frames, the feature point is considered to have stable motion characteristics and is marked as a stable motion feature point.

[0053] For each obstacle region, the percentage of stable motion feature points within it is calculated. When the number of stable feature points exceeds 75% of the total number of feature points in that region, the obstacle is classified as a stable motion obstacle; otherwise, it is classified as an unstable motion obstacle.

[0054] For a stable moving obstacle, a trajectory prediction model based on Kalman filtering is established. The obstacle's position, velocity, and acceleration are used as state variables, and observation data from the past 0.5-second time window are input into the model. The system noise covariance matrix is ​​dynamically adjusted based on the uncertainty of feature point tracking, while the observation noise covariance matrix is ​​determined based on the reliability of feature matching. This model predicts the obstacle's trajectory within the next 0.5 to 1 second, with the confidence level of the prediction gradually decreasing over time.

[0055] Specifically, the system noise covariance matrix assesses uncertainty by calculating the tracking error of feature points. First, for each feature point within the obstacle region, the Euclidean distance between its predicted and actual observed positions is calculated between adjacent time steps as the tracking error. The mean square tracking error of all feature points is then calculated as the baseline value for the system noise variance of the position component. For the velocity component, the time derivative of the position noise variance (the difference between adjacent time steps divided by the time interval) is calculated as its noise variance; similarly, the time derivative of the velocity noise variance is used for the acceleration component. When the tracking error exceeds a preset threshold, the diagonal elements of the system noise covariance matrix are increased proportionally. When the tracking error is small across multiple consecutive frames, the system noise covariance value is gradually decreased.

[0056] The observation noise covariance matrix is ​​used to determine the reliability of the observation data based on the quality assessment of feature matching. First, fixed-size image patches are extracted around each feature point, and the normalized cross-correlation coefficients of these patches between adjacent frames are calculated. The cross-correlation coefficients range from -1 to 1 and are converted to a reliability index of 0 to 1 using a mapping function. When the reliability is higher than 0.8, the observation noise variance is set to 50% of the baseline value; when the reliability is between 0.5 and 0.8, the baseline value remains unchanged; when the reliability is lower than 0.5, the observation noise variance is increased to twice the baseline value. For feature points that may be occluded, an additional 50% of the observation noise variance is added.

[0057] For unstable moving obstacles, a multi-hypothesis state estimation model is constructed. First, multiple possible motion mode hypotheses are established for each unstable moving obstacle, including uniform motion, uniformly accelerated motion, and turning motion. Each motion mode corresponds to an independent Kalman filter, and an interactive multi-model algorithm is introduced to calculate the probability weights of each motion mode.

[0058] Specifically, three basic motion modes are defined: uniform motion (CV model), uniformly accelerated motion (CA model), and turning motion (CT model).

[0059] For the CV model, the state vector includes position and velocity; the CA model adds an acceleration component; and the CT model includes position, velocity, and angular velocity. Each motion mode uses a different state transition equation: the CV model assumes constant velocity, the CA model assumes constant acceleration, and the CT model considers the effect of angular velocity on the trajectory. Initially, the probability weights for all three models are set to 1 / 3.

[0060] Each motion pattern corresponds to an independent Kalman filter. The state transition matrix and observation matrix are configured according to the characteristics of each motion pattern. For the CV model, a 4-dimensional state vector [x, y, vx, vy] is used; for the CA model, a 6-dimensional state vector [x, y, vx, vy, ax, ay] is used; and for the CT model, a 5-dimensional state vector [x, y, v, θ, ω] is used, where θ is the motion direction angle and ω is the angular velocity. The system noise and observation noise covariance matrices of each filter employ the same dynamic adjustment mechanism as for stable moving obstacles.

[0061] At each time step, a model interaction process is first performed. The model transition probability matrix is ​​calculated, and this matrix is ​​used to calculate the mixture probability. The state estimates and covariances of each model are then fused based on these mixture probabilities, serving as the initial values ​​for the next time step prediction. Each model is independently predicted and updated, yielding its respective likelihood function value. Based on the likelihood function and the model transition probabilities, the probability weights of each model are updated.

[0062] Specifically, the model transition probability matrix is ​​a 3×3 matrix that records the transition probabilities between three motion modes: CV, CA, and CT. Within a 2-second time window (40 frames), if the weight of a certain motion mode increases from below 0.2 to above 0.6, it is recorded as a mode transition. The transition probability is obtained by counting the number of transitions from the starting mode to the ending mode and dividing by the total number of transitions. If a certain transition does not occur at all within 40 frames, its transition probability is set to 0.05. For example, if the CV to CA transition occurs 5 times, and the total number of transitions is 20, then the transition probability is set to 0.25.

[0063] When calculating the mixture probability, the transition probability matrix is ​​multiplied by the model weights from the previous time step. Assuming the CV model weights were 0.7 at the previous time step and the transition probability from CV to CA was 0.25, the corresponding mixture probability would be 0.175. The state vectors of the three models are then weighted and averaged according to the calculated mixture probability. The same weighting is used for fusing the covariance matrix.

[0064] At each new time step (0.05 seconds), the three models independently perform predictions. The CV model assumes the velocity remains constant; the CA model extrapolates using the current acceleration value; and the CT model calculates the circular trajectory based on the current angular velocity. After obtaining new observation data, the Mahalanobis distance between the observed and predicted values ​​is calculated. If the prediction error is less than 1 times the observation standard deviation, the likelihood function is set to 1; if the error is between 1 and 3 times the standard deviation, the likelihood function is calculated according to a Gaussian distribution; if the error exceeds 3 times the standard deviation, the likelihood function value is set to 0.01.

[0065] The likelihood function value is multiplied by the model transition probability to obtain the new weights for each model. To avoid abrupt weight changes, the change in weights in a single update is limited to no more than 30% of the original weights. For example, if the original weight of a model is 0.4, the new weight's range is limited to 0.28 to 0.52. Simultaneously, it is ensured that the weight of each model is not lower than 0.05. When the difference in the likelihood function values ​​of the three models is less than 10%, the original weight distribution remains unchanged. Finally, all weights are normalized so that their sum is 1.

[0066] S3: Based on the obstacle index data, the obstacles are grouped, a spatial relationship map between the obstacles is established, and the obstacle state is estimated hierarchically based on the spatial relationship map. Combined with the prediction model, the trajectory of obstacles at different occlusion levels is tracked.

[0067] Obstacles are grouped based on occlusion status indicators, and the occlusion relationship between each pair of obstacles is calculated, including indicators such as the proportion of occlusion area and spatial position. An occlusion relationship is determined if the overlapping area of ​​the projected regions of two obstacles exceeds 20% of the area of ​​the smaller object, or if the distance in the depth direction is less than 2 meters. All obstacles with direct or indirect occlusion relationships are grouped together to form occlusion groups.

[0068] Within each occlusion group, a spatial relationship graph is constructed, where nodes represent obstacles and edges represent occlusion relationships. For each edge, the spatial positional relationship between the occluding and occluded objects, the size and location of the occluded area, and the temporal attribute of the occlusion occurrence are recorded. To improve the stability of the graph structure, a temporal consistency check is introduced: if an occlusion relationship is observed in multiple consecutive frames, the trend of its attribute changes is verified, and relationships with abnormal changes are eliminated.

[0069] When performing occlusion relationship analysis, the existence of multiple occlusions is determined based on the occlusion depth index. Specifically, the image is divided into grid cells for depth index calculation. The degree of occlusion is assessed by analyzing the number of depth discontinuities and the degree of depth jumps within each grid cell. A depth jump exceeding 0.5 meters within a grid cell is counted as a depth discontinuity. Adjacent grid cells are grouped; if the depth difference exceeds 0.3 meters, they are classified into the same occlusion region, and the maximum occlusion depth value is taken as the index for that region.

[0070] The presence of multiple occlusions is determined based on occlusion depth metrics and feature point behavior patterns. If the occlusion depth metric of a region is greater than 2, and the feature points within that region exhibit a multi-layered disappearance-reappearance pattern, multiple occlusions are identified. Specifically, the following criteria must be met: the time interval between feature point disappearances and reappearances shows obvious layering (time difference greater than 0.2 seconds), the movement speed difference between feature points at different layers is significant (exceeding 30%), and the depth map in that region exhibits a step-like change (at least two significant depth jumps, each greater than 0.8 meters). When these conditions are met simultaneously, the region is identified as having multiple occlusions, and the depth occlusion processing mode is activated.

[0071] For regions identified as having multiple occlusions, a directed occlusion graph is constructed. Nodes in the graph represent obstacles, and edges represent occlusion relationships. The weight of each edge is determined by three factors: the proportion of overlapping area of ​​the occluded regions (calculating the overlapping portion of the projected area of ​​the occluded object), the degree of depth difference (the magnitude of depth jumps between adjacent obstacles), and the duration of occlusion (the number of frames in which occlusion relationships are continuously detected). The occlusion level of each obstacle is determined through a hierarchical traversal of the occlusion graph; the level number is equal to the number of edges pointing to that node plus one.

[0072] Different tracking strategies are employed for different occlusion levels. The foreground layer (occluded layer) uses a complete set of feature points for tracking; the intermediate layer uses a prediction-verification framework for tracking, maintaining the tracking state using motion prediction when feature points temporarily disappear, and correcting the position when feature points reappear. The background layer (occluded layer) mainly relies on a motion model for prediction, while maintaining a historical trajectory database of feature points, and matching based on trajectory similarity when feature points reappear.

[0073] The tracking results of each layer are input into the state estimator to establish inter-layer constraints, optimize the state estimation results of each layer, fuse the information of each layer, reconstruct the complete scene structure, including synthesizing the spatial positional relationships of each layer, restoring the geometry of the occluded area, and ensuring the temporal consistency of the scene.

[0074] Specifically, the state estimator uses an extended Kalman filter framework to maintain an 8-dimensional state vector for each obstacle, which includes position coordinates (x, y), velocity (vx, vy), acceleration (ax, ay), and shape parameters (length l, width w).

[0075] In the foreground layer, complete observation data is acquired every 50ms, including obstacle contour point cloud extracted from the image, center position, and current velocity. These observations are matched with state predictions, Kalman gain is calculated, and then the state vector and covariance matrix are updated. At this point, the observation noise covariance matrix R is set to a small value (0.1 m position error, 0.2 m / s velocity error) to make full use of high-quality observation data.

[0076] When an obstacle enters the intermediate occlusion layer, the observation data becomes partially visible. If the number of visible feature points is greater than 50%, the same direct tracking method as the foreground layer is used, but the uncertainty of the position component in the observation noise covariance matrix R is increased to 0.2 m, and the uncertainty of the velocity component is increased to 0.3 m / s. If the number of visible feature points is less than or equal to 50%, it is considered an unstable obstacle, and the state estimation relies on a multi-hypothesis state estimation model for prediction. The state estimator adjusts the process noise covariance Q, and the uncertainty of the position component is increased to 0.3 m, and the uncertainty of the velocity component is increased to 0.5 m / s.

[0077] For completely occluded obstacles in the background layer, they are considered unstable obstacles by default, and state estimation relies on a multi-hypothesis state estimation model for prediction. The motion trajectory is recorded within 2 seconds before the occlusion occurs, and key motion parameters are extracted: the standard deviation of acceleration σ, the standard deviation of the rate of change of steering angle θ, and the root mean square error (RMSE) of position deviation.

[0078] If σ is less than 0.3 m / s² and θ is less than 3° / s, it is determined to be a stable linear motion. The uncertainty of the position component in the process noise covariance Q is set to 0.3 m and the uncertainty of the velocity component is set to 0.5 m / s.

[0079] If σ is between 0.3 and 0.8 m / s² and θ is less than 5° / s, it is considered to be a steady variable speed motion; the uncertainty of the position component in the process noise covariance Q is set to 0.4 m and the uncertainty of the velocity component is set to 0.8 m / s.

[0080] If θ is greater than 5° / s, it is classified as a turning motion, and the uncertainty of the position component in the process noise covariance Q is set to 0.5 m, and the uncertainty of the velocity component is set to 1.0 m / s.

[0081] If the occlusion lasts for more than 1 second, the trajectory extrapolation protection mechanism is activated to limit the cumulative error of the predicted state to no more than 30% of the last visible position.

[0082] When performing multi-layer state fusion, the inter-layer constraints include: spatial location constraints, motion consistency constraints, and scale consistency constraints.

[0083] Spatial location constraints ensure that the minimum distance between obstacles at different levels is no less than 0.5 meters. When the predicted location violates this constraint, the state estimates of each level are adjusted through quadratic programming. Motion consistency constraints require that the acceleration change of obstacles at adjacent levels does not exceed 2 m / s², which is achieved by adding a velocity smoothing term to the objective function. For scale consistency constraints, the scale change during occlusion is limited to within ±10% by maintaining a priori library of shapes from historical observations.

[0084] These constraints are integrated into a 15-frame sliding time window for joint optimization. The objective function J comprises four terms: state prediction error Jp, observation residual Jo, inter-layer constraints Jc, and temporal smoothing Js, i.e., J = w1Jp + w2Jo + w3Jc + w4Js, where the weight coefficients are dynamically adjusted based on the reliability of the observation data from each layer. Iterative optimization is performed using gradient descent until the objective function converges or the maximum number of iterations is reached.

[0085] Regarding the weighting coefficients, the weight w1 of the state prediction error term increases with the integrity of the observed data, adjusting within the range of [0.2, 0.6]. w1 increases by 0.05 for every 10% decrease in observed data quality. The weight w2 of the observation residual term is positively correlated with the reliability of the observed data, ranging from [0.15, 0.45], and is determined by the success rate of feature point tracking and the level of observation noise. The weight w3 of the inter-layer constraint term is allocated based on the strength of the occlusion relationship, located in the range of [0.1, 0.3], and is proportional to the proportion of occlusion area and the magnitude of depth jumps. The weight w4 of the temporal smoothing term is controlled within the range of [0.05, 0.25], adjusted according to the temporal stability of the state estimation; when a sudden state change is detected, the weight value increases with the magnitude of the change. Normalization ensures that the sum of all weights is 1, and the weight configuration is updated in each estimation period, enabling the objective function to better adapt to different observation conditions and occlusion scenarios.

[0086] Based on the optimized state estimation, a spatial distribution map of obstacles at each layer is constructed. For occluded areas, their geometric contours are recovered by combining the shape parameters at the last visible state and the current predicted state. The reconstruction results are processed using a Gaussian smoothing filter (window width of 5 frames) to eliminate jitter caused by state transitions and ensure the temporal continuity of scene reconstruction.

[0087] S4: Based on the motion trajectory prediction model, the multi-hypothesis state estimation model, and the estimation results of the obstacle state, construct an obstacle avoidance priority decision tree, and control the robot's motion direction and speed according to the output of the decision tree.

[0088] When a stable moving obstacle is detected, the output of the motion trajectory prediction model is used as the primary prediction source, while the prediction result of the multi-hypothesis state estimation model is used as auxiliary verification. If the difference between the two models' predictions is less than 15% of the predicted distance, a weighted average method is used for fusion, with a weight ratio of 7:3; if the difference is greater than 15%, the result of the motion trajectory prediction model is retained. When an unstable moving obstacle is detected, the prediction of the multi-hypothesis state estimation model is used as the primary source, while the motion trajectory prediction model is used as a constraint. If the change in the motion trajectory predicted by the multi-hypothesis model exceeds twice the predicted value of the motion trajectory model, the change is limited to within twice the predicted value.

[0089] For obstacles with occlusion relationships, a prediction confidence score is established based on the obstacle state inference results. When the obstacle is in the foreground layer, the prediction confidence baseline score is set to 0.9; when it is in the middle layer, the baseline score drops to 0.7; and when it is in the background layer, the baseline score further drops to 0.5. If the occlusion depth index is greater than 2, the confidence score of the corresponding layer decreases by 0.1 for each additional layer of occlusion depth. When the confidence score is below 0.3, the obstacle is marked as a high-risk target.

[0090] When constructing the obstacle avoidance priority decision tree, the 360-degree space around the robot is divided into 8 sectors. Within each sector, three layers of child nodes are established based on the predicted state of the obstacle: the first layer represents the obstacle's motion type (stable / unstable), the second layer represents the occlusion state (foreground / middle / background), and the third layer represents the prediction confidence (high / medium / low). When multiple obstacles exist within a sector, the obstacle with the highest risk score is selected as the representative of that sector. The risk score is calculated as follows: base risk value (inversely proportional to distance) × motion type coefficient (1 for stable, 1.5 for unstable) × occlusion level coefficient (1 for foreground, 1.3 for middle, 1.6 for background) ÷ prediction confidence.

[0091] For each sector branch, the base weight is proportional to the risk score of the obstacle represented by that sector. If the risk score of a sector changes by more than 30% within 5 consecutive frames, that sector is marked as an unstable region. For unstable regions, an additional 25% weight is added. If a new occlusion relationship is detected, the weight of the relevant sector gradually increases to 1.5 times its original value over 2 seconds. If the prediction confidence is below 0.5, the weight of the corresponding branch is increased by 40%.

[0092] If the motion continuity index indicates a sudden change in the motion of an obstacle, the weight of the sector containing that obstacle is increased by 20%. If the occlusion state changes, the weight of the sector containing the new occluder increases by 30%, and the weight of the sector containing the occluded object increases by 50%. If the occlusion depth increases, the weight of the relevant sector increases by 25% for each additional layer of depth.

[0093] Regarding the recovery process after weight adjustment, the weight values ​​will gradually decrease from their maximum adjusted values ​​and gradually return to the baseline level following an exponential law. During the recovery phase after weight adjustment, the initial weight value is set to the maximum adjusted value, and then the weight value gradually decreases at fixed intervals according to a certain multiplier. The decay rate is determined by the risk complexity of the scenario. For example, in high-risk scenarios, the decay multiplier will be smaller, making the weight recovery slower; while in low-risk scenarios, the decay multiplier will be larger, making the weight return to the baseline level faster.

[0094] Specifically, the recovery process is calculated from the start of the weight adjustment. The change in weight value at each moment is proportional to the difference between the current weight value and the baseline weight value, and the proportionality factor decreases as the scenario risk increases. For example, when the occlusion level is high, the time required for weight recovery will be significantly longer, while when the occlusion level is low, the recovery speed will be significantly faster. All weight adjustment effects decay exponentially to the baseline level within 3 seconds.

[0095] The robot's movement is controlled based on the weight distribution output by the decision tree. The sector with the smallest weight is selected as the obstacle avoidance direction, and the upper speed limit is set based on the distance to the nearest obstacle in that direction. Speed ​​adjustments are made smoothly, with acceleration limited to within 0.5 m / s². For directional control, the deviation between the target heading and the current heading is calculated, and the maximum turning angular velocity is set to 45° / s. Proportional control is used for steering adjustments, with the proportional coefficient inversely proportional to the speed.

[0096] The base safety distance is set at 1.5 meters. If the uncertainty in the predicted obstacle movement increases, the safety distance is increased by 0.2 meters for every 10% increase in uncertainty. If a new layer of obstruction is detected, an additional 0.3 meters of safety distance is added for each new layer. If the relative speed exceeds 2 m / s, an additional 0.25 meters of safety distance is added for every 1 m / s increase. For obstacles with a prediction confidence level below 0.5, an additional 0.5 meters of safety distance is added.

[0097] In summary, the vision-based robot obstacle avoidance control system and method based on the embodiments of the present invention have been elucidated. Through multi-scale feature extraction, improved feature point tracking algorithms, the definition of occlusion depth indices and multi-layer occlusion analysis, and a comprehensive design combining a multi-hypothesis state estimation model and an obstacle avoidance priority decision tree, it significantly improves the robot's obstacle detection, occlusion handling, and obstacle avoidance decision-making capabilities in complex dynamic environments. It can accurately determine the motion characteristics and spatial relationships of obstacles at different scales and occlusion levels, and dynamically adjust obstacle avoidance strategies, enabling the robot to achieve efficient and stable path planning and navigation even in high-risk, multi-occluded, and unstable motion scenarios.

Claims

1. A robot obstacle avoidance control method based on vision recognition, characterized in that, include: A multi-scale edge feature map of the image in front of the robot is acquired, a spatiotemporal feature point sequence is constructed on the multi-scale edge feature map, and obstacle index data is calculated. Based on the obstacle index data, obstacles are divided into stable moving obstacles and unstable moving obstacles. A motion trajectory prediction model is established for the stable moving obstacles, and a multi-hypothesis state estimation model is established for the unstable moving obstacles. Based on the obstacle index data, obstacles are grouped, a spatial relationship map between obstacles is established, and the obstacle state is estimated hierarchically based on the spatial relationship map. Combined with the prediction model, the trajectory of obstacles at different occlusion levels is tracked. Based on the motion trajectory prediction model, the multi-hypothesis state estimation model, and the estimation results of the obstacle state, an obstacle avoidance priority decision tree is constructed, and the robot's motion direction and speed are controlled according to the output of the decision tree.

2. The robot obstacle avoidance control method based on vision recognition according to claim 1, characterized in that, Based on the stable moving obstacle, a motion trajectory prediction model based on Kalman filtering is adopted, using position, velocity, and acceleration as state variables, using observation data from the past time period to predict the trajectory, and dynamically adjusting the noise covariance matrix according to the reliability of feature point tracking and feature matching.

3. The robot obstacle avoidance control method based on vision recognition according to claim 2, characterized in that, Based on the aforementioned multi-hypothesis state estimation model, three basic motion modes—uniform motion, uniformly accelerated motion, and turning motion—are assumed. Each mode corresponds to an independent Kalman filter, and the probability weights of each mode are dynamically adjusted using an interactive multi-model algorithm.

4. The robot obstacle avoidance control method based on vision recognition according to claim 1, characterized in that, The obstacle index data includes motion continuity index, occlusion status index, and occlusion depth index.

5. The robot obstacle avoidance control method based on vision recognition according to claim 4, characterized in that, Based on the occlusion status index, obstacles are grouped to determine whether occlusion relationships exist. Obstacles with occlusion relationships are divided into occlusion groups. Within each occlusion group, a spatial association graph is constructed to record the occlusion attributes between obstacles. Abnormal relationships are eliminated through a temporal consistency check.

6. The robot obstacle avoidance control method based on vision recognition according to claim 5, characterized in that, Based on the occlusion depth index, the analysis of multiple occlusion situations is conducted. When the depth jump in the area is significant and the feature points exhibit a multi-layered disappearance-reappearance pattern, it is determined to be a multiple occlusion area and the processing mode is activated. By establishing a directed occlusion relationship graph and combining the occlusion area ratio, depth jump amplitude, and occlusion duration, the occlusion level of the obstacle is determined.

7. The robot obstacle avoidance control method based on vision recognition according to claim 6, characterized in that, Differentiated tracking strategies are adopted based on different occlusion levels: the foreground layer uses complete feature point tracking, the intermediate layer uses a prediction-validation framework to handle feature point disappearance and position correction, and the background layer relies on motion model prediction and uses historical trajectory matching for feature points.

8. The robot obstacle avoidance control method based on vision recognition according to claim 7, characterized in that, The tracking results of the differentiated tracking strategy are input into the extended Kalman filter framework. Combined with interlayer constraints, the obstacle state is optimized and estimated to reconstruct the complete scene structure and restore the geometric contour of the occluded area.

9. The robot obstacle avoidance control method based on vision recognition according to claim 8, characterized in that, An obstacle avoidance priority decision tree is constructed based on the motion trajectory prediction model, the multi-hypothesis state estimation model, and the estimation results of the obstacle state. The obstacle avoidance priority decision tree divides the surrounding environment into sectors, calculates risk scores based on the obstacle's movement type, occlusion status, and confidence level, dynamically adjusts sector weights, and selects the direction with the lowest weight for obstacle avoidance.

10. A robot obstacle avoidance control system based on vision recognition, comprising controlling robot obstacle avoidance using the robot obstacle avoidance control method based on vision recognition as described in any one of claims 1-9, characterized in that, include: The acquisition module collects image data in front of the robot, constructs a spatiotemporal feature point sequence, and calculates obstacle index data; The prediction module classifies obstacles into obstacle types based on the obstacle index data and constructs prediction models for each type. The classification module groups obstacles based on the obstacle index data, establishes a spatial relationship map between obstacles, performs hierarchical estimation of obstacle status, and combines a prediction model to track the trajectory of obstacles at different occlusion levels. The control module constructs an obstacle avoidance priority decision tree based on the prediction model and the estimation results of the obstacle state, and controls the robot's movement direction and speed according to the output of the decision tree.

Images