Vision-based robot arm obstacle avoidance method and system

Abstract

The application relates to the technical field of mechanical arm obstacle avoidance, and particularly discloses a mechanical arm obstacle avoidance method and system based on vision. The method comprises the following steps: collecting environmental vision data to identify auxiliary potential obstacles and construct an attribute matching list; combining task real-time execution stage features, improving an ideal solution sorting algorithm to construct an auxiliary adaptation degree evaluation model, determining an auxiliary target obstacle after sorting, designing an auxiliary scheme and extracting a path planning core constraint; positioning an auxiliary function node and an initial path, and outputting a cooperative motion trajectory through quadratic programming optimization; and calculating a failure judgment coefficient based on the auxiliary adaptation degree and a demand achievement rate, and performing local reconstruction or updating of the trajectory according to scenes. The system comprises an obstacle identification module, a path planning module, a trajectory adaptation module and a trajectory updating module. The application converts obstacles from obstacle avoidance objects into auxiliary resources, which is beneficial to improving the task efficiency of the mechanical arm.

Description

Technical Field

[0001] This invention relates to the field of obstacle avoidance technology for robotic arms, and specifically to a vision-based obstacle avoidance method and system for robotic arms. Background Technology

[0002] Robotic arms are widely used in industrial settings for tasks such as material handling, precision assembly, and tool changing. Their obstacle avoidance capabilities during movement directly affect task execution efficiency and safety.

[0003] Currently, most mainstream obstacle avoidance technologies for robotic arms focus on "avoiding obstacle interference." They collect environmental information through visual sensors (such as monocular cameras and lidar) or force sensors, identify obstacles that may interfere with the workspace of the robotic arm, and then use path planning algorithms to generate motion trajectories that bypass the obstacles, ensuring that the end effector of the robotic arm maintains a safe distance from the obstacles.

[0004] However, traditional methods simply define obstacles as "interference objects," focusing only on their hindering effect on the movement of robotic arms, without considering the auxiliary value that the physical characteristics (such as surface flatness, structural stability, and material hardness) of obstacles in the workspace (such as assembled parts, idle supports, and material boxes) may possess. For example, idle supports with flat surfaces can serve as temporary material storage platforms, and tooling fixture protrusions with stable structures can serve as positioning references for precision assembly. If these potential auxiliary capabilities can be utilized, the reliance on additional auxiliary equipment (such as dedicated temporary storage platforms and positioning fixtures) can be reduced, which is conducive to reducing the deployment cost in industrial scenarios.

[0005] Therefore, the present invention provides a vision-based obstacle avoidance method and system for robotic arms. Summary of the Invention

[0006] The purpose of this invention is to provide a vision-based obstacle avoidance method and system for robotic arms to solve the aforementioned background problems.

[0007] The objective of this invention can be achieved through the following technical solutions:

[0008] A vision-based obstacle avoidance method for robotic arms includes the following steps:

[0009] Collect the physical characteristics of environmental obstacles and the robot arm's task parameters, identify potential obstacles, and establish an attribute matching list for potential obstacles;

[0010] Based on the attribute matching list, the potential obstacles to be assisted are evaluated to obtain the assisted fit degree and the fit degree is prioritized to determine the assisted scheme for the target obstacle and the core constraints of the robotic arm path planning.

[0011] Based on the core constraints, the auxiliary target node of the auxiliary target obstacle is located and embedded into the robot arm motion path planning, and the robot arm cooperative motion trajectory of the auxiliary target obstacle is output.

[0012] Based on the cooperative motion trajectory, the auxiliary target obstacle is evaluated for failure, a failure judgment index is obtained, and it is determined whether the auxiliary target obstacle has failed. If it fails, the trajectory is reconstructed locally to update the cooperative motion trajectory.

[0013] As a further aspect of the present invention: the method for identifying the auxiliary potential obstacle is as follows:

[0014] The physical characteristic parameters are integrated to construct a physical parameter group of the physical characteristic parameters;

[0015] Integrate the task parameters of the robotic arm to construct a task parameter group;

[0016] Construct a feature-demand correlation rule base by acquiring historical interaction data from industrial scenarios;

[0017] The physical parameter set and the task parameter set are input into the feature-requirement correlation rule base for parameter matching processing to obtain the comprehensive matching degree.

[0018] Based on the comprehensive matching degree, the obstacle is subjected to auxiliary potential screening to identify auxiliary potential obstacles.

[0019] As a further aspect of the present invention, the adaptation evaluation is performed as follows:

[0020] Based on the attribute matching list, physical feature parameters and real-time execution stage features of M auxiliary potential obstacles are extracted to construct a multi-dimensional decision matrix;

[0021] Feature deviation analysis was performed based on the multidimensional decision matrix to obtain the positive and negative feature distances of each auxiliary potential obstacle.

[0022] The adaptation evaluation is performed based on the positive and negative feature distances of each auxiliary potential obstacle to obtain the auxiliary adaptation degree.

[0023] As a further aspect of the present invention, the method for performing the feature deviation analysis is as follows:

[0024] Extract the physical feature parameters of each auxiliary potential obstacle from the multidimensional decision matrix, and construct the potential feature vector of each auxiliary potential obstacle;

[0025] Obtain the positive ideal solutions for all physical characteristic parameters and construct a positive eigenvector; obtain the negative ideal solutions for all physical characteristic parameters and construct a negative eigenvector.

[0026] Obtain the correction factor coefficient, calculate the Euclidean distance between the potential feature vector and the positive feature vector of each auxiliary potential obstacle, and sum the Euclidean distance calculation result with the correction factor coefficient to obtain the positive feature distance of the auxiliary potential obstacle.

[0027] Calculate the Euclidean distance between the potential feature vector and the negative feature vector of each auxiliary potential obstacle, and sum it with the correction factor coefficient to obtain the negative feature distance of the auxiliary potential obstacle.

[0028] As a further aspect of the present invention, the method for obtaining the correction factor coefficient is as follows:

[0029] Obtain the historical multidimensional decision matrix and calculate the absolute value of the Pearson correlation coefficient between any two physical feature parameters as the feature coupling coefficient;

[0030] Calculate the absolute difference between any two physical feature parameters of each auxiliary potential obstacle in the current multidimensional decision matrix, multiply it by the feature coupling coefficient of the two features, and take the square root to obtain the coupling correction factor of the two sets of features.

[0031] As a further aspect of the present invention: the method for outputting the cooperative motion trajectory of the robotic arm is as follows:

[0032] The auxiliary target node is used as a necessary node in the path of the robotic arm, and the initial motion path of the robotic arm is generated; at the same time, it avoids ordinary obstacles and the non-auxiliary area of ​​the auxiliary target obstacle.

[0033] The initial motion path of the robotic arm is optimized using a quadratic programming algorithm to obtain the cooperative motion trajectory of the robotic arm.

[0034] As a further aspect of the present invention, the initial motion path is generated in the following way:

[0035] The path planning is divided into two stages, with the state of the robotic arm joint corresponding to the auxiliary function node as the necessary node;

[0036] Phase 1: Construct a first random tree, with the root node being the starting joint state of the robotic arm and the target node being the necessary node. During sampling, some samples are directly taken from the necessary node and some samples are taken from Gaussian distribution points with the mean of the necessary node. The random tree is expanded through the sampling strategy and collision detection is performed to generate the first-stage path.

[0037] Second stage: Construct a second random tree with the required nodes as the root nodes, and the target node is the final joint state of the robotic arm. Expand the random tree through a sampling strategy to generate the second stage path;

[0038] The initial motion path is obtained by merging the two-stage paths.

[0039] As a further aspect of the present invention, the method for performing the auxiliary failure assessment is as follows:

[0040] Obtain the failure determination coefficient, and based on the failure determination coefficient and the comprehensive matching degree, perform scenario-based failure determination processing on the auxiliary target obstacle.

[0041] As a further aspect of the present invention, the failure determination coefficient is obtained as follows:

[0042] Calculate the deviation rate between the current auxiliary fit and the initial auxiliary fit to obtain the fit change value;

[0043] Obtain the key requirements for each stage, calculate the requirement achievement rate for each stage, and the average requirement achievement rate for all stages.

[0044] The failure determination coefficient is obtained by multiplying the average demand fulfillment rate with the adaptation variation value.

[0045] A vision-based obstacle avoidance system for robotic arms includes the following modules:

[0046] Obstacle recognition module: used to collect the physical characteristics of environmental obstacles and the robot arm's task parameters, identify potential obstacles, and establish an attribute matching list of potential obstacles;

[0047] Path planning module: Based on the attribute matching list, it is used to evaluate the adaptation of potential obstacles to obtain the adaptation degree and sort the adaptation degree priority to determine the assistance scheme for the target obstacle and the core constraints of the robotic arm path planning.

[0048] Trajectory adaptation module: Based on core constraints, it is used to locate the auxiliary target node of the auxiliary target obstacle and embed the robot arm motion path planning, and output the robot arm cooperative motion trajectory of the auxiliary target obstacle;

[0049] The trajectory update module performs auxiliary failure assessment on the auxiliary target obstacle based on the cooperative motion trajectory, obtains the failure judgment index, and determines whether the auxiliary target obstacle has failed. If it has failed, it performs local trajectory reconstruction to update the cooperative motion trajectory.

[0050] The beneficial effects of this invention are:

[0051] (1) Collect multi-dimensional visual data, extract the physical features of obstacles, and construct a feature-demand correlation rule library by associating the robot arm task parameters. This is beneficial for selecting objects with auxiliary potential from the obstacles in the robot arm workspace, transforming the simple interference objects in the traditional obstacle avoidance scenario into usable potential resources, and providing basic support for subsequent robot arm task assistance. Furthermore, filtering objects in non-operation areas by the robot arm workspace boundary helps optimize obstacle recognition efficiency, while prioritizing the screening of static obstacles, thus ensuring the stability of the auxiliary function.

[0052] (2) Based on the physical feature parameters in the attribute matching list and combined with the characteristics of the real-time execution stage of the task, the ideal solution sorting algorithm is improved to construct the fit evaluation model, which is conducive to dynamically measuring the fit of different auxiliary potential obstacles with the current task, and determining the auxiliary target obstacles that fit the task requirements through priority sorting; the designed auxiliary scheme and the extracted path planning core constraints can combine the auxiliary requirements with the path planning logic, while taking into account the real-time state changes of the task, and improving the flexibility of fit evaluation and constraint setting.

[0053] (3) Based on the core constraints, the optimal region of auxiliary function is extracted from the auxiliary target obstacle and the auxiliary node is determined by the point cloud region growth algorithm. The RRT* algorithm is used to set the node as a necessary node on the path, which is beneficial for the robotic arm to use the obstacle to complete the task during the movement. At the same time, it avoids the non-auxiliary areas of ordinary obstacles and auxiliary target obstacles, taking into account both obstacle avoidance and assistance requirements. Subsequently, the initial path is optimized by the quadratic planning algorithm, which helps to improve the smoothness of the robotic arm's movement trajectory, reduce oscillations or sudden changes during the movement, and provide support for the stable execution of the task.

[0054] (4) Based on the initial and current auxiliary adaptation degree and the achievement rate of key stage requirements, calculate the failure judgment coefficient, and perform failure judgment and processing on the auxiliary target obstacle in different scenarios. When it is not failed, continuously monitor the status. When it fails, perform trajectory reconstruction or auxiliary requirement cancellation according to whether the robot arm has reached the obstacle position. This is conducive to dealing with the dynamic changes of the auxiliary target obstacle in the task execution process and reducing the task interruption caused by obstacle failure. In the trajectory update process, reuse the algorithm and parameters of the previous steps, which helps to ensure the continuity and adaptability of trajectory adjustment and realize the continuous advancement of the robot arm task in the dynamic environment. Attached Figure Description

[0055] The invention will now be further described with reference to the accompanying drawings.

[0056] Figure 1 This is a flowchart of a vision-based obstacle avoidance method for robotic arms according to the present invention;

[0057] Figure 2 This is a flowchart of the method for assisting in the identification of potential obstacles in this invention;

[0058] Figure 3 This is a functional block diagram of a vision-based obstacle avoidance system for robotic arms in this invention. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] Example 1

[0061] Please see Figure 1 As shown, the present invention is a vision-based obstacle avoidance method for robotic arms, comprising the following steps:

[0062] S1. Collect the physical characteristics of environmental obstacles and the robot arm's task parameters, identify potential obstacles, and establish an attribute matching list for potential obstacles;

[0063] The method for collecting the physical characteristics of environmental obstacles and the robot arm's task parameters is as follows:

[0064] In some embodiments, based on the RGB-D camera and 3D structured light sensor installed on the robotic arm, raw visual data of the environment (including RGB images, depth maps, and point cloud data) is collected. A point cloud segmentation algorithm is used in conjunction with the workspace boundary of the robotic arm (a pre-set X / Y / Z axis motion range) to mark the set of object points in the point cloud that are located within the workspace and are not part of the robotic arm's own structure as obstacles.

[0065] It should be noted that environmental obstacles are various objects within the robotic arm's workspace that may interfere with its movement trajectory, including tooling fixture protrusions, assembled parts, idle supports, material boxes, etc. The physical characteristics of environmental obstacles directly determine whether they have the potential to assist the robotic arm in completing its tasks, and the installation location of the vision acquisition equipment must cover the entire workspace of the robotic arm. During the obstacle identification process, filtering by the boundaries of the robotic arm's workspace can exclude objects outside the operating area, improving recognition efficiency. Static obstacles are given priority in the assistance potential screening due to their stability advantages, while dynamic obstacles are not considered at this time.

[0066] The raw visual data of the marked obstacles are preprocessed to extract physical feature parameters;

[0067] For example, the preprocessing method is as follows: Gaussian filtering is used to denoise the RGB image to remove environmental noise interference; the point cloud data is registered using the Iterative Closest Point (ICP) algorithm to fuse multiple frame point clouds into a complete point cloud model in a unified coordinate system, providing a data foundation for feature extraction;

[0068] Among them, physical characteristic parameters include: surface morphology characteristics, structural stability characteristics, spatial occupancy characteristics, and material property characteristics of obstacles;

[0069] Surface morphology features include: obtaining the surface flatness of the obstacle through point cloud curvature analysis, and constructing a gray-level co-occurrence matrix based on the RGB image of the obstacle, and calculating the surface texture entropy of the obstacle based on the gray-level co-occurrence matrix;

[0070] Structural stability features include: calculating the volume of obstacles by clustering obstacles from point clouds, and analyzing static or dynamic properties based on position changes in three consecutive RGB images. For example, obstacles whose position changes less than 5 mm in two consecutive frames are considered static obstacles.

[0071] Spatial occupancy features include: obtaining the three-dimensional coordinates of the obstacle, the size of the obstacle's circumscribed cube, and the angle of the normal vector through a depth map boundary detection algorithm;

[0072] Material property features include: identifying the material type of obstacles based on the color histogram and reflectance of the RGB image, such as plastic, metal or wood, and surface hardness estimates, such as a reflectance higher than 0.7 for hard surfaces and a reflectance lower than 0.3 for soft surfaces;

[0073] The physical characteristic parameters are integrated to construct a physical parameter group of the physical characteristic parameters;

[0074] Synchronously obtain the current task parameters of the robotic arm from the interface of the robotic arm control system or task scheduling system;

[0075] The task parameters include: task type, task stage (e.g., positioning stage, operation stage, or temporary storage stage), and tool accuracy requirements.

[0076] It should be noted that the task types include: material handling that requires a temporary storage platform, precision assembly that requires a positioning reference, tool replacement that requires a tool placement location, and environmental isolation that requires an anti-interference barrier.

[0077] Integrate the task parameters of the robotic arm to construct a task parameter group;

[0078] Establish a feature-requirement correlation rule base, input the physical parameter group and task parameter group into the feature-requirement correlation rule base for parameter matching processing, and obtain the comprehensive matching degree;

[0079] The parameter matching process is performed as follows:

[0080] The core requirement keywords are extracted from the task parameter group. For each core requirement keyword, the threshold and weight coefficient of the corresponding physical feature parameter are matched in the feature-requirement correlation rule base. It is then determined whether the physical feature parameters in the physical parameter group meet the threshold of the physical feature parameter.

[0081] All physical characteristic parameters are dimensionless.

[0082] If the physical feature parameter meets the threshold of the physical feature parameter, then obtain the weight coefficient of the corresponding physical feature parameter, calculate the percentage of the absolute deviation between the physical feature parameter and the threshold of the physical feature parameter, and obtain the matching margin value.

[0083] If the physical feature parameters do not meet the threshold of the physical feature parameters, then the weight coefficient is 0;

[0084] The local matching degree of each physical feature parameter is obtained by multiplying each physical feature parameter with its corresponding weight coefficient.

[0085] The sum of the local matching degrees of all physical feature parameters is calculated to obtain the comprehensive matching degree.

[0086] Understandably, the method for establishing the feature-requirement correlation rule base is as follows: based on 500 sets of historical interaction data from industrial scenarios covering multiple tasks such as material handling, precision assembly, tool replacement, and environmental isolation, including task requirements, obstacle physical characteristics, and auxiliary effects, corresponding physical feature parameter thresholds are matched for each task requirement keyword (e.g., temporary tool requirements correspond to surface flatness ≤ 0.5mm / m, hard surface (reflectivity > 0.7), and volume > 0.1m). 3 The correlation strength of each physical feature parameter to the auxiliary effect is calculated by using the Pearson correlation coefficient, and corresponding weight coefficients are assigned (such as surface smoothness weight 0.6 and material hardness weight 0.4). After verification by those skilled in the art, a feature-requirement correlation rule library containing three-dimensional structured entries of task requirement keywords, physical feature parameter thresholds and weight coefficients is formed, which supports dynamic updates.

[0087] Based on the comprehensive matching degree, auxiliary potential obstacles are screened to identify auxiliary potential obstacles;

[0088] like Figure 2 As shown, the method for screening auxiliary potential is as follows: set a preset threshold for comprehensive matching degree (determined based on the lowest matching degree that meets the auxiliary effect standard in historical data, such as 0.7), compare the comprehensive matching degree of the obstacle with the threshold, if the comprehensive matching degree is ≥ the preset comprehensive matching threshold, then mark the obstacle as an auxiliary potential obstacle, if the comprehensive matching degree is < the preset comprehensive matching threshold, then no processing is performed;

[0089] Organize the numbering, physical characteristic parameters and comprehensive matching degree of the potential assistive obstacles, perform dimensionless processing on the physical characteristic parameters, and establish an attribute matching list of the potential assistive obstacles.

[0090] The role of screening for potential obstacles is as follows:

[0091] Function 1: To screen out objects with auxiliary potential from obstacles in the workspace of the robotic arm, transforming simple interference objects in traditional obstacle avoidance scenarios into usable potential resources, reducing reliance on additional auxiliary equipment such as dedicated temporary storage platforms and positioning fixtures, and adapting to the diverse task requirements of industrial scenarios.

[0092] Secondly, by filtering based on comprehensive matching degree to form an attribute matching list, the physical characteristics and comprehensive matching degree of the potential obstacles to be assisted are clearly defined. This provides a precise and focused object basis for subsequent assistance fit assessment, selection of target obstacles to be assisted, and extraction of core constraints for path planning, reducing invalid calculations and improving the pertinence and efficiency of the overall process.

[0093] S2. Based on the attribute matching list, the potential obstacles to be assisted are evaluated to obtain the assisted fit degree and the fit degree priority is sorted to determine the assisted scheme for the target obstacle and the core constraints of the robotic arm path planning.

[0094] The method for assessing the fitness of assistive potential obstacles and prioritizing the fitness scores is as follows:

[0095] In some embodiments, the execution phase characteristics of the current task are obtained from the control system of the robotic arm;

[0096] The characteristics of the execution phase include: phase type (e.g., positioning phase, operation phase, temporary storage phase), key phase requirements, and remaining phase duration.

[0097] For example, key requirements for each stage include: the positioning stage requires a position change of ≤5mm, the operation stage requires stable posture support, and the temporary storage stage requires a load-bearing capacity of ≥8kg.

[0098] Based on the physical feature parameters of the attribute matching list and the features of the real-time execution stage, an evaluation model for the assist fit is constructed by combining the ideal solution sorting algorithm, and the assist fit of the assist potential obstacle is output.

[0099] The method for constructing the evaluation model for auxiliary fit is as follows:

[0100] S201. Based on the attribute matching list, extract the physical feature parameters and real-time execution stage features of M auxiliary potential obstacles, and construct a multi-dimensional decision matrix.

[0101] Preferably, M auxiliary potential obstacles are used as evaluation objects, and N core physical feature parameters (such as surface flatness error, volume, three-dimensional coordinates, reflectivity) are selected for each obstacle to construct an original decision matrix of M rows and N columns;

[0102] In the original decision matrix, each value represents the physical characteristic parameter of the corresponding auxiliary potential obstacle (e.g., the surface flatness error of an obstacle is 0.3 mm / m).

[0103] The original decision matrix is ​​standardized by dividing each physical feature parameter by the square root of the sum of the squares of all auxiliary potential obstacles of the same type. The larger the standardized value, the closer it is to the ideal state.

[0104] The standardized original decision matrix is ​​labeled as a multidimensional decision matrix;

[0105] Based on the task execution phase type (location phase, operation phase, temporary storage phase) and the key requirements of the phase, the positive ideal solution (optimal eigenvalue) and negative ideal solution (worst eigenvalue) of each physical characteristic parameter are dynamically defined.

[0106] S202. Based on the multidimensional decision matrix, perform feature deviation analysis to obtain the positive and negative feature distances of each auxiliary potential obstacle.

[0107] Preferably, the method for performing feature deviation analysis is as follows:

[0108] Obtain Z historical multidimensional decision matrices, and calculate the absolute value of the Pearson correlation coefficient between any two physical feature parameters as the feature coupling coefficient;

[0109] Preferably, Z=500;

[0110] Calculate the absolute difference between any two physical feature parameters of each auxiliary potential obstacle in the current multidimensional decision matrix, multiply it by the feature coupling coefficient of the two features, and take the square root to obtain the coupling correction factor of the two sets of features.

[0111] Extract the physical feature parameters of each auxiliary potential obstacle from the multidimensional decision matrix, and construct the potential feature vector of each auxiliary potential obstacle;

[0112] Obtain the positive ideal solution for all physical characteristic parameters and construct a positive feature vector;

[0113] Obtain the negative ideal solution of all physical characteristic parameters and construct the negative feature vector;

[0114] Calculate the Euclidean distance between the potential feature vector and the positive feature vector of each auxiliary potential obstacle, and sum the Euclidean distance calculation result with the correction factor coefficient to obtain the positive feature distance of the auxiliary potential obstacle;

[0115] Calculate the Euclidean distance between the potential feature vector and the negative feature vector of each auxiliary potential obstacle, and sum it with the correction factor coefficient to obtain the negative feature distance of the auxiliary potential obstacle.

[0116] S203. Based on the positive and negative feature distances of each auxiliary potential obstacle, an adaptation evaluation is performed to obtain the auxiliary adaptation degree.

[0117] The preferred method for performing adaptation evaluation is as follows:

[0118] The positive and negative characteristic distances of each auxiliary potential obstacle are summed to obtain the total characteristic distance.

[0119] The ratio of negative feature distance to the sum of feature distances is calculated to obtain the auxiliary fit.

[0120] Sort the obstacles in descending order according to their assist fit scores, and select the obstacle with the highest assist fit score as the target obstacle.

[0121] Extract the physical feature parameters of the auxiliary target obstacle and the task requirements from the attribute matching list, and design an auxiliary solution to obtain the following:

[0122] For example, the method for designing auxiliary solutions is as follows:

[0123] If the target obstacle is used to temporarily store the tool, then the coordinates of the temporary storage location should be clearly defined (calculated based on the obstacle's three-dimensional coordinates and the optimal area for surface flatness, such as selecting the coordinates of the center area with the smallest surface flatness error (X=1.2m, Y=0.8m, Z=0.5m)). The upper load-bearing limit should also be specified (based on volume and material hardness estimation, such as for metal materials with a volume of 0.2m). 3 The maximum load capacity is 10kg) and the placement posture (the angle between the tool end and the normal vector of the obstacle surface is ≤2°).

[0124] If used as a positioning reference, the coordinates of the reference point (selecting the center point of the area with the lowest surface texture entropy of the obstacle), the positioning accuracy compensation value (based on the reverse correction of surface flatness error, such as compensating +0.1mm when the flatness error is 0.2mm / m), and the posture constraints of the robotic arm end (the deviation of the angle from the obstacle normal vector ≤1°) should be specified.

[0125] Extracting the core constraints for robotic arm path planning based on auxiliary schemes;

[0126] The core constraints include: spatial position constraints, attitude constraints, load constraints, and obstacle avoidance-assistance coordination constraints.

[0127] For example, spatial position constraints: the end effector of the robotic arm needs to reach the auxiliary target node (1.2m, 0.8m, 0.5m), with a coordinate deviation ≤ 0.004m;

[0128] Attitude constraints: The angle between the end effector of the robotic arm and the normal vector of the surface of the auxiliary target obstacle is ≤1°;

[0129] Load constraint: The weight of the material placed at the end of the robotic arm is less than or equal to the maximum load-bearing capacity of the auxiliary target obstacle (8 kg).

[0130] Obstacle avoidance - assisted collaborative constraint: When the robotic arm moves close to the assisted area, the distance between it and the non-assisted area and ordinary obstacles is ≥0.05m.

[0131] Example 2

[0132] Please see Figure 1 As shown, the present invention is a vision-based obstacle avoidance method for robotic arms, which further includes the following steps:

[0133] S3. Based on the core constraints, locate the auxiliary target node of the auxiliary target obstacle and embed it into the robot arm motion path planning, and output the robot arm cooperative motion trajectory of the auxiliary target obstacle;

[0134] The method for locating auxiliary functional nodes of the target obstacle and embedding them into the robotic arm motion path planning, based on core constraints, is as follows:

[0135] Based on the core constraints and physical feature parameters of the auxiliary target obstacle, a point cloud region growth algorithm is used to extract the optimal region and non-auxiliary region for auxiliary functions, providing auxiliary target points and non-auxiliary regions for path planning.

[0136] Preferably, based on core constraints (such as surface flatness error, normal vector angle deviation, etc.) and physical feature parameter thresholds, the center of the obstacle point cloud is used as the seed point, and the points in the neighborhood that meet the constraints (such as surface flatness ≤ 0.5 mm / m and normal vector angle ≤ 2°) are selected through the point cloud region growth algorithm, and the points are grown and merged to form a continuous optimal auxiliary function region.

[0137] The center position of the optimal area for auxiliary functions is taken as the auxiliary target node;

[0138] Regions that are not grown (such as protrusions or small areas that do not meet the constraints) are marked as non-auxiliary regions;

[0139] The RRT* algorithm, based on auxiliary target nodes, treats these nodes as essential nodes in the robotic arm's path and generates the initial motion path. Simultaneously, it avoids ordinary obstacles (i.e., non-auxiliary target obstacles) and the non-auxiliary areas of auxiliary target obstacles.

[0140] As will be understood by those skilled in the art, the RRT* algorithm uses the auxiliary target node as a necessary node in the robot arm's path to generate the initial motion path of the robot arm in the following way:

[0141] The path planning is divided into stages, and a biased sampling strategy is used to select the necessary nodes for each stage, and the necessary nodes are used as intermediate points of the path.

[0142] Preferably, in the first stage: construct a random tree T1, where the root node of the random tree is the initial joint state of the robotic arm. The target node is the joint state corresponding to the necessary node. An improved random sampling strategy is used to generate sampling points during the growth of the random tree T1. ;

[0143] 20% probability of direct sampling (Forced exploration of necessary nodes), with a 50% probability that the sampling follows a Gaussian distribution. The point (biased) Neighborhood, where the mean is The standard deviation σ = 0.1 rad, determined based on the robotic arm joint angular resolution of 0.01 rad, ensures that the sampling points are clustered in the area. Within a range of ±0.3 rad, there is a 30% probability of uniform sampling in the joint space;

[0144] The random tree T1 is expanded by sampling strategy. During the expansion process, collision detection is performed on the generated candidate nodes to ensure that the non-auxiliary areas of ordinary obstacles and auxiliary target obstacles are avoided (judged by the distance between the node's corresponding robotic arm state and the obstacle being greater than or equal to the safety threshold).

[0145] The collision detection safety threshold is 0.05 meters (based on the size of the robotic arm end effector and the safety redundancy setting in the industrial scenario, consistent with the safety distance of 'obstacle avoidance-assisted cooperative constraint' in S4).

[0146] When growth can directly connect The node (i.e., the node and) If the Euclidean distance in the joint space is ≤0.1 rad (satisfying motion continuity), then stop the first phase and record. to All nodes form the first-stage path P1: , for to The middle node;

[0147] Phase Two: Construct a new random tree T2 for the root node, with the target node being the final target joint state of the robotic arm. ;

[0148] When T2 grows, it can be directly connected For nodes (joint space distance ≤ 0.1 rad), the same collision detection is performed during expansion to avoid two types of obstacles. The second phase is then stopped, and the path of the second phase is recorded. ;

[0149] ,in for and intermediate nodes;

[0150] The first-stage path and the second-stage path are merged to form the initial motion path;

[0151] The method for outputting the cooperative motion trajectory of the auxiliary target obstacle-type robotic arm is as follows:

[0152] Simultaneously, a quadratic programming algorithm is used to optimize the initial motion path of the robotic arm, improve the smoothness of the initial motion path, and obtain the cooperative motion trajectory of the robotic arm.

[0153] It is understandable that the method of optimizing the initial motion path of the robotic arm using the quadratic programming algorithm is as follows: When optimizing using the quadratic programming algorithm, the goal is to minimize the sum of the squares of the second derivatives of the joint angles of the robotic arm (to reduce acceleration abrupt changes and improve trajectory smoothness). The joint angle sequence of the initial motion path is used as the optimization variable. At the same time, the core constraints extracted by S2 (such as the coordinate deviation of the auxiliary target node ≤ 0.004 meters, the angle between the end effector and the normal vector of the auxiliary obstacle ≤ 0.5°), the range of motion of the robotic arm joints (±180°), and the obstacle avoidance requirements (distance from the obstacle ≥ safety threshold) are embedded as constraints. The optimized joint angle, velocity, and acceleration parameter sequence is obtained by solving the problem, and finally a robotic arm cooperative motion trajectory that meets the requirements of assistance and smoothness is formed.

[0154] S4. Based on the cooperative motion trajectory, perform auxiliary target obstacle failure assessment, obtain failure judgment index and determine whether the auxiliary target obstacle has failed. If it has failed, perform local trajectory reconstruction processing to update the cooperative motion trajectory.

[0155] The method for assessing the failure of an auxiliary target obstacle based on cooperative motion trajectory is as follows:

[0156] In some embodiments, the underlying data is explicitly evaluated, including:

[0157] Initial Assist Fit: The initial value of the assist fit of the target assist obstacle in S2;

[0158] Preset comprehensive matching threshold: auxiliary potential screening standard in S1 (0.7);

[0159] Key requirements for each stage: Key requirements for the current task in S2 (e.g., positioning stage: coordinate reference error ≤ 0.05mm, normal vector angle deviation ≤ 2°; temporary storage stage: load capacity ≥ 8kg, surface flatness ≤ 0.5mm / m).

[0160] Acquire dynamic evaluation data of the robotic arm during its cooperative motion trajectory, including:

[0161] Current assist fit: obtained based on the S2-based assist fit evaluation model;

[0162] Current overall matching degree: calculated based on the feature-demand correlation rule base of S1;

[0163] Calculate the deviation rate between the current auxiliary fit and the initial auxiliary fit to obtain the fit change value, which is used to reflect the fit decay or improvement trend;

[0164] It should be noted that the deviation rate between the current assist fit and the initial assist fit is calculated as follows: Fit variation value = (current assist fit - initial assist fit) / initial assist fit × 100%, where 'initial assist fit' is the initial assist fit value of the target assist obstacle in S2, and 'current assist fit' is the real-time value recalculated based on the S2 evaluation model.

[0165] For example: if the initial fit is 0.82 and the current fit is 0.58, the fit change value = (0.58-0.82) / 0.82×100%≈-29.27% ​​(a negative value indicates a decrease in fit).

[0166] For each key requirement in S2, calculate the requirement achievement rate of each key requirement in each stage, and the average requirement achievement rate of all key requirements in all stages;

[0167] For example, the achievement rate is calculated based on the actual execution data collected visually: Achievement rate = (stage key requirement threshold / actual deviation value) × 100% (achievement rate = 100% when deviation value ≤ threshold).

[0168] For example: In the positioning stage, the coordinate reference error threshold is 0.05mm, the actual deviation is 0.07mm, and the achievement rate is (0.05 / 0.07)×100%≈71.43%; the normal vector angle deviation threshold is 2°, the actual deviation is 1.5°, and the achievement rate is 100%; the average achievement rate of key requirements across all stages is calculated as: (71.43%+100%) / 2≈85.72%;

[0169] The failure determination coefficient is obtained by multiplying the average demand fulfillment rate with the adaptation variation value.

[0170] Based on the failure determination coefficient and comprehensive matching degree, the auxiliary target obstacle is subjected to scenario-specific failure determination processing:

[0171] If the failure judgment coefficient is ≥0 (adaptability has not decayed or although it has decayed, the requirement achievement rate is extremely low, resulting in a non-negative coefficient): continuously monitor the real-time changes of the current auxiliary adaptability and requirement achievement rate, and recalculate the failure judgment coefficient every 500ms.

[0172] If the failure judgment coefficient is less than 0 (adaptability decays and requirements are not fully met), and the auxiliary target obstacle is still an auxiliary potential obstacle (current comprehensive matching degree ≥ preset comprehensive matching threshold 0.7), then calculate the degree of closeness between the current comprehensive matching degree and the preset comprehensive matching threshold to obtain the degree of closeness value;

[0173] For example, the closeness value is calculated as (current overall matching degree - preset overall matching threshold) / preset overall matching threshold × 100%, which reflects the closeness between the overall matching degree and the threshold.

[0174] If the failure determination coefficient is less than 0, and the auxiliary target obstacle does not meet the conditions for an auxiliary potential obstacle (current comprehensive matching degree < preset comprehensive matching threshold 0.7), then the auxiliary target obstacle is determined to be a failure: Analyze whether the robotic arm reaches the position of the auxiliary target obstacle on the cooperative motion trajectory:

[0175] The method for determining whether the robotic arm has reached the position of the auxiliary target obstacle on the cooperative motion trajectory is as follows: Based on the path point coordinates of the S3 cooperative motion trajectory, compare the current end-effector coordinates of the robotic arm with the auxiliary function node coordinates of the auxiliary target obstacle (such as node coordinates (1.2m, 0.8m, 0.5m) in S3). If the distance between the two is >0.1m (2.5 times the path node deviation threshold in S3), it is determined that the position of the auxiliary target obstacle has not been reached.

[0176] If the location of the auxiliary target obstacle is not reached: the Top2-3 candidate auxiliary potential obstacle list of S2 is called, and the current auxiliary fitness of the candidate obstacles is recalculated based on the ideal solution ranking method model of S2 (reusing the real-time physical feature parameters of S1). The candidate obstacle with the highest current fitness is selected as the new auxiliary target. The B-spline interpolation algorithm + quadratic programming optimization of S3 is reused to embed the auxiliary function node of the new auxiliary target into the original trajectory, reconstruct the "original effective path segment - new auxiliary node path - target point path", and output the updated cooperative motion trajectory.

[0177] If the location of the auxiliary target obstacle has been reached: cancel the auxiliary requirements (such as positioning reference, temporary support) located at the auxiliary target obstacle, remove the auxiliary interaction nodes in the original trajectory that depend on the obstacle based on the trajectory reconstruction logic of S3, and replan the path from the current position to the task target point using the RRT* algorithm to ensure that the distance between the path and the failed obstacle is ≥ the safe bypass distance of S3 (such as 0.65m) to reduce collisions;

[0178] Output the final coordinated motion trajectory (including adjusted path point coordinates, joint angles, and motion parameters) to ensure uninterrupted task execution and adaptability to current auxiliary conditions.

[0179] Example 3

[0180] Please see Figure 3 As shown, the present invention is a vision-based obstacle avoidance system for robotic arms, comprising the following modules:

[0181] Obstacle recognition module: used to collect the physical characteristics of environmental obstacles and the robot arm's task parameters, identify potential obstacles, and establish an attribute matching list of potential obstacles;

[0182] Path planning module: Based on the attribute matching list, it is used to evaluate the adaptation of potential obstacles to obtain the adaptation degree and sort the adaptation degree priority to determine the assistance scheme for the target obstacle and the core constraints of the robotic arm path planning.

[0183] Trajectory adaptation module: Based on core constraints, it is used to locate the auxiliary target node of the auxiliary target obstacle and embed the robot arm motion path planning, and output the robot arm cooperative motion trajectory of the auxiliary target obstacle;

[0184] The trajectory update module performs auxiliary failure assessment on the auxiliary target obstacle based on the cooperative motion trajectory, obtains the failure judgment index, and determines whether the auxiliary target obstacle has failed. If it has failed, it performs local trajectory reconstruction to update the cooperative motion trajectory.

[0185] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the present invention should still fall within the scope of the present invention.

Claims

1. A vision-based obstacle avoidance method for robotic arms, characterized in that: Includes the following steps: Collect the physical characteristics of environmental obstacles and the robot arm's task parameters, identify potential obstacles, and establish an attribute matching list for potential obstacles; The method for identifying the assistive potential obstacle is as follows: The physical characteristic parameters are integrated to construct a physical parameter group of the physical characteristic parameters; Integrate the task parameters of the robotic arm to construct a task parameter group; Construct a feature-demand correlation rule base by acquiring historical interaction data from industrial scenarios; The physical parameter set and the task parameter set are input into the feature-requirement correlation rule base for parameter matching processing to obtain the comprehensive matching degree. Based on the comprehensive matching degree, auxiliary potential obstacles are screened to identify auxiliary potential obstacles; Based on the attribute matching list, the potential obstacles to be assisted are evaluated to obtain the assisted fit degree and the fit degree is prioritized to determine the assisted scheme for the target obstacle and the core constraints of the robotic arm path planning. Based on the core constraints, the auxiliary target node of the auxiliary target obstacle is located and embedded into the robot arm motion path planning, and the robot arm cooperative motion trajectory of the auxiliary target obstacle is output. Based on the cooperative motion trajectory, the auxiliary target obstacle is evaluated for failure, a failure judgment index is obtained, and it is determined whether the auxiliary target obstacle has failed. If it fails, the trajectory is reconstructed locally to update the cooperative motion trajectory.

2. The vision-based obstacle avoidance method for a robotic arm according to claim 1, characterized in that: The adaptation assessment is performed as follows: Based on the attribute matching list, physical feature parameters and real-time execution stage features of M auxiliary potential obstacles are extracted to construct a multi-dimensional decision matrix; Feature deviation analysis was performed based on the multidimensional decision matrix to obtain the positive and negative feature distances of each auxiliary potential obstacle. The adaptation evaluation is performed based on the positive and negative feature distances of each auxiliary potential obstacle to obtain the auxiliary adaptation degree.

3. The vision-based obstacle avoidance method for a robotic arm according to claim 2, characterized in that: The method for performing the feature deviation analysis is as follows: Extract the physical feature parameters of each auxiliary potential obstacle from the multidimensional decision matrix, and construct the potential feature vector of each auxiliary potential obstacle; Obtain the positive ideal solutions for all physical characteristic parameters and construct a positive eigenvector; obtain the negative ideal solutions for all physical characteristic parameters and construct a negative eigenvector. Obtain the correction factor coefficient, calculate the Euclidean distance between the potential feature vector and the positive feature vector of each auxiliary potential obstacle, and sum the Euclidean distance calculation result with the correction factor coefficient to obtain the positive feature distance of the auxiliary potential obstacle. Calculate the Euclidean distance between the potential feature vector and the negative feature vector of each auxiliary potential obstacle, and sum it with the correction factor coefficient to obtain the negative feature distance of the auxiliary potential obstacle.

4. The vision-based obstacle avoidance method for a robotic arm according to claim 3, characterized in that: The correction factor coefficients are obtained as follows: Obtain the historical multidimensional decision matrix and calculate the absolute value of the Pearson correlation coefficient between any two physical feature parameters as the feature coupling coefficient; Calculate the absolute difference between any two physical feature parameters of each auxiliary potential obstacle in the current multidimensional decision matrix, multiply it by the feature coupling coefficient of the two features, and take the square root to obtain the coupling correction factor of the two sets of features.

5. The vision-based obstacle avoidance method for a robotic arm according to claim 1, characterized in that: The method for outputting the cooperative motion trajectory of the robotic arm is as follows: The auxiliary target node is used as a necessary node in the path of the robotic arm, and the initial motion path of the robotic arm is generated; at the same time, it avoids ordinary obstacles and the non-auxiliary area of ​​the auxiliary target obstacle. The initial motion path of the robotic arm is optimized using a quadratic programming algorithm to obtain the cooperative motion trajectory of the robotic arm.

6. The vision-based obstacle avoidance method for a robotic arm according to claim 5, characterized in that: The initial motion path is generated as follows: The path planning is divided into two stages, with the state of the robotic arm joint corresponding to the auxiliary function node as the necessary node; Phase 1: Construct a first random tree, with the root node being the starting joint state of the robotic arm and the target node being the necessary node. During sampling, some samples are directly taken from the necessary node and some samples are taken from Gaussian distribution points with the mean of the necessary node. The random tree is expanded through the sampling strategy and collision detection is performed to generate the first-stage path. Second stage: Construct a second random tree with the required nodes as the root nodes, and the target node is the final joint state of the robotic arm. Expand the random tree through a sampling strategy to generate the second stage path; The initial motion path is obtained by merging the two-stage paths.

7. The vision-based obstacle avoidance method for a robotic arm according to claim 1, characterized in that: The method for conducting the aforementioned auxiliary failure assessment is as follows: Obtain the failure determination coefficient, and based on the failure determination coefficient and the comprehensive matching degree, perform scenario-based failure determination processing on the auxiliary target obstacle.

8. The vision-based obstacle avoidance method for a robotic arm according to claim 7, characterized in that: The failure determination coefficient is obtained as follows: Calculate the deviation rate between the current auxiliary fit and the initial auxiliary fit to obtain the fit change value; Obtain the key requirements for each stage, calculate the requirement achievement rate for each stage, and the average requirement achievement rate for all stages. The failure determination coefficient is obtained by multiplying the average demand fulfillment rate with the adaptation variation value.

9. A vision-based obstacle avoidance system for a robotic arm, used to implement the vision-based obstacle avoidance method for a robotic arm as described in any one of claims 1-8, characterized in that: Includes the following modules: Obstacle recognition module: used to collect the physical characteristics of environmental obstacles and the robot arm's task parameters, identify potential obstacles, and establish an attribute matching list of potential obstacles; Path planning module: Based on the attribute matching list, it is used to evaluate the adaptation of potential obstacles to obtain the adaptation degree and sort the adaptation degree priority to determine the assistance scheme for the target obstacle and the core constraints of the robotic arm path planning. Trajectory adaptation module: Based on core constraints, it is used to locate the auxiliary target node of the auxiliary target obstacle and embed the robot arm motion path planning, and output the robot arm cooperative motion trajectory of the auxiliary target obstacle; The trajectory update module performs auxiliary failure assessment on the auxiliary target obstacle based on the cooperative motion trajectory, obtains the failure judgment index, and determines whether the auxiliary target obstacle has failed. If it has failed, it performs local trajectory reconstruction to update the cooperative motion trajectory.

Images