Industrial robot three-dimensional vision guidance AI precision control method

Abstract

The application discloses an industrial robot three-dimensional vision guidance AI precision control method, relates to the technical field of intelligent control of industrial robots, and comprises the following steps: acquiring and preprocessing three-dimensional vision point cloud data to construct a dynamic environment model. Based on the model and task constraints, a plurality of candidate robot motion trajectories are generated, and a pre-trained trajectory evaluation network is used to score the quality of each trajectory, according to which the optimal trajectory is screened out and is decomposed into joint control instructions. When the robot is driven to execute, actual data and model prediction data are synchronously collected and compared online to generate an execution deviation. According to the deviation, subsequent control instructions are corrected in real time through an online compensation algorithm, and a closed loop is formed. Through intelligent trajectory optimization and model prediction-based feedforward compensation, the application improves the decision-making intelligence, control precision and overall robustness of the robot in a complex dynamic environment.

Description

Technical Field

[0001] This invention belongs to the field of intelligent control technology for industrial robots, specifically a three-dimensional vision-guided AI-based precise control method for industrial robots. Background Technology

[0002] Currently, mainstream technologies for 3D vision-guided industrial robots typically include relatively independent perception, planning, and control modules. At the planning level, the system plans a trajectory based on a single pose estimation of the target object by the vision system, combined with a pre-defined optimization objective based on kinematics and deterministic mathematical models. This method often generates a single trajectory, whose "optimality" heavily relies on a pre-defined quantization function, making it difficult to effectively accommodate and balance the various complex, unstructured, and potentially conflicting implicit requirements inherent in the task. This results in insufficient overall quality and adaptability of the planning results in dynamic or unstructured scenarios.

[0003] At the control level, existing methods primarily rely on closed-loop feedback control of the robot's joint positions to track planned trajectory commands. This control mode is essentially a delayed adjustment of the internal state, and its feedback information does not include perception and response to actual changes in the external environment during execution. The system typically relies on visual perception before the action begins, lacking a mechanism for proactively adjusting unexecuted actions based on real-time environmental feedback during execution. Therefore, when facing dynamic environments or with system errors, the final execution accuracy and task success rate are difficult to reliably guarantee. Summary of the Invention

[0004] This invention aims to solve at least one of the technical problems existing in the prior art;

[0005] Therefore, this invention proposes a three-dimensional vision-guided AI-based precise control method for industrial robots, including:

[0006] Acquire raw point cloud data collected by a 3D vision sensor and preprocess it to generate scene understanding data;

[0007] Based on scene understanding data, a dynamic environment model is constructed to describe the pose of the target object and its spatial relationship with the robot's end effector.

[0008] Using a dynamic environment model and combining it with preset robot task constraints, multiple candidate robot motion trajectories are generated through a spatial search algorithm, forming a candidate trajectory set;

[0009] The spatiotemporal features of each candidate robot motion trajectory in the candidate trajectory set are extracted, input into a pre-trained trajectory evaluation network for calculation, and the trajectory quality score of each candidate robot motion trajectory is output.

[0010] Based on the trajectory quality score, the optimal robot motion trajectory is selected from the candidate trajectory set and decomposed into a sequence of joint space control instructions that the robot can execute;

[0011] The joint space control command sequence is sent to the robot controller to drive the robot to perform motion, and the actual execution data is collected synchronously during the motion.

[0012] The actual execution data is compared online with the predicted data from the dynamic environment model to generate trajectory execution deviation data;

[0013] Based on trajectory execution deviation data, the subsequent joint space control command sequence is corrected in real time through an online compensation algorithm to achieve closed-loop precise control.

[0014] Furthermore, raw point cloud data acquired by a 3D vision sensor is preprocessed to generate scene understanding data, including:

[0015] The system acquires raw point cloud data containing the target object collected by a 3D vision sensor, and preprocesses the raw point cloud data to obtain denoised 3D point cloud data.

[0016] Scene semantic segmentation is performed on the denoised 3D point cloud data to identify the structured 3D information of the target object and its surrounding environment, and to generate scene understanding data.

[0017] The process of acquiring raw point cloud data containing the target object collected by a 3D vision sensor and preprocessing the raw point cloud data to obtain denoised 3D point cloud data specifically involves:

[0018] Receive raw point cloud data collected by a 3D vision sensor in the robot's workspace, wherein the raw point cloud data includes target object point cloud, background point cloud and noise point cloud;

[0019] Perform statistical filtering on the original point cloud data to remove spatially discrete outlier noise point clouds;

[0020] Voxel mesh downsampling is performed on the filtered point cloud data to reduce the point cloud density while preserving the geometric features of the scene;

[0021] Smoothing is performed on the downsampled point cloud data to obtain denoised 3D point cloud data with smooth surface and significantly reduced noise.

[0022] Furthermore, scene semantic segmentation is performed on the denoised 3D point cloud data to identify the structured 3D information of the target object and its surrounding environment, generating scene understanding data, specifically:

[0023] The denoised 3D point cloud data is input into the 3D semantic segmentation neural network model;

[0024] A semantic label is assigned to each point in the point cloud data using a 3D semantic segmentation neural network model. The semantic label includes at least the target object, the bearing plane, the protective fence, and the irrelevant background.

[0025] The point cloud data is clustered into different instance objects based on semantic tags, and the point cloud clusters of each instance object are extracted.

[0026] The system calculates the 3D bounding box, centroid coordinates, and surface normal vectors of the point cloud clusters representing the target object, and extracts the plane equation parameters of the bearing plane. Together, they constitute structured 3D information describing the scene and generate scene understanding data.

[0027] Furthermore, based on scene understanding data, a dynamic environment model describing the pose of the target object and its spatial relationship with the robot's end effector is constructed, specifically as follows:

[0028] Analyze the scene to understand the 3D bounding box and centroid coordinates of the target object in the data, and calculate the six-degree-of-freedom pose of the target object in the current robot base coordinate system;

[0029] Calculate the pose of the robot's end effector in the initial state based on the robot's kinematic model;

[0030] Calculate the relative position and orientation relationship between the target object pose and the initial pose of the robot end effector, and construct the initial spatial relationship matrix;

[0031] By combining the 3D map information of the robot's workspace, the target object's pose, the initial spatial relationship matrix, and the geometric information of environmental obstacles are integrated to construct a dynamic environment model, which is updated as the robot moves.

[0032] Furthermore, utilizing a dynamic environment model and combining it with pre-defined robot task constraints, a spatial search algorithm generates multiple candidate robot motion trajectories, forming a candidate trajectory set, specifically:

[0033] Obtain the initial pose of the robot's end effector and the target grasping pose from the dynamic environment model;

[0034] Based on preset robot task constraints, including joint angular velocity limits, avoidance of singular configurations, and shortest path preference, random sampling is performed within the robot configuration space.

[0035] Based on sampling points, a path connection algorithm is used to plan a continuous path connecting the starting pose and the target grasping pose while avoiding obstacles in the dynamic environment model.

[0036] For each continuous path, time parameter interpolation is performed to generate a sequence of the end-effector's Cartesian spatial position and orientation as a function of time, which is a candidate robot trajectory.

[0037] Repeat the planning and generation steps to generate multiple candidate robot motion trajectories that differ in geometric path or time parameters, forming a candidate trajectory set.

[0038] Furthermore, the spatiotemporal features of each candidate robot trajectory in the candidate trajectory set are extracted and input into a pre-trained trajectory evaluation network for calculation, outputting a trajectory quality score for each candidate robot trajectory, specifically:

[0039] For each candidate robot motion trajectory in the candidate trajectory set, extract its trajectory smoothness features, minimum distance to obstacle features, joint motion coordination features, and expected execution time features to form the spatiotemporal feature vector of the candidate robot motion trajectory.

[0040] The spatiotemporal feature vectors corresponding to each candidate robot motion trajectory are sequentially input into a pre-trained trajectory evaluation network;

[0041] The trajectory evaluation network is a multilayer perceptron structure. It outputs a scalar value that represents the overall quality of the trajectory through forward propagation calculation, which is the trajectory quality score of the candidate robot's motion trajectory.

[0042] Furthermore, based on the trajectory quality score, the optimal robot motion trajectory is selected from the candidate trajectory set and decomposed into a sequence of joint space control commands that the robot can execute, specifically:

[0043] Compare the trajectory quality scores of all candidate robot motion trajectories in the candidate trajectory set, and select the candidate robot motion trajectory with the highest trajectory quality score as the optimal robot motion trajectory;

[0044] By using robot inverse kinematics, the end-effector pose at each time point on the optimal robot motion trajectory is calculated into the corresponding robot joint angle values.

[0045] According to the preset control cycle, the continuous sequence of joint angle values ​​is discretized into a series of joint angle target commands arranged in chronological order, forming a joint space control command sequence.

[0046] Furthermore, the joint space control command sequence is sent to the robot controller to drive the robot to perform movements, and the actual execution data is collected synchronously during the movements, specifically:

[0047] The joint space control command sequence is sent one by one to the real-time control loop of the robot controller;

[0048] The robot controller drives the motors of each joint of the robot to move according to the joint space control command sequence;

[0049] Meanwhile, the actual angle values ​​of each joint are read in real time through the joint encoder of the robot body, and the actual pose point cloud data of the robot end tool and target object are collected in real time through the three-dimensional vision sensor.

[0050] The actual angle value and the actual pose point cloud data are synchronously encapsulated and output as the actual execution data.

[0051] Furthermore, the actual execution data is compared online with the predicted data from the dynamic environment model to generate trajectory execution deviation data, specifically:

[0052] The actual pose point cloud data of the robot's end effector is extracted from the actual execution data, and its actual centroid pose is calculated.

[0053] Obtain the predicted pose of the robot's end effector at the current moment from the dynamic environment model;

[0054] Calculate the difference between the actual centroid pose and the predicted pose in each of the position and attitude components;

[0055] The difference, along with the difference between the actual joint angle value in the actual execution data and the corresponding instruction in the joint space control instruction sequence, are integrated into trajectory execution deviation data.

[0056] Furthermore, based on the trajectory execution deviation data, the subsequent joint space control command sequence is corrected in real time through an online compensation algorithm to achieve closed-loop precise control, specifically as follows:

[0057] The trajectory execution deviation data is input into the online compensation algorithm, which is constructed based on the proportional-integral-differential principle.

[0058] The online compensation algorithm calculates the compensation amount used to correct subsequent joint angle target commands based on the historical and current values ​​of trajectory execution deviation data.

[0059] The compensation amount is superimposed on the subsequent joint angle target commands in the joint space control command sequence that has not yet been executed, to generate the corrected joint space control command sequence.

[0060] The corrected joint space control command sequence is sent to the robot controller to replace the original command sequence, so as to drive the robot to complete subsequent movements and form a closed-loop precision control process of perception, decision-making, execution and correction.

[0061] Compared with the prior art, the beneficial effects of the present invention are:

[0062] A pre-trained trajectory evaluation network is used to score and filter multiple candidate trajectories. This network, through offline learning, integrates and quantifies various complex, unstructured, and difficult-to-model optimization objectives involved in task execution. Its decision-making is based on data-driven induction from historical successes and failures. This transforms trajectory selection from a single, pre-defined mathematical cost function into an intelligent decision-making process based on multi-dimensional implicit knowledge evaluation. The system can therefore autonomously identify trajectories with superior overall performance from numerous feasible solutions, improving decision-making quality and task adaptability in uncertain environments.

[0063] This method constructs an online feedforward compensation mechanism based on dynamic environment model prediction. During robot execution, the system continuously compares the collected actual execution data with the predicted data generated by the dynamic environment model in real time, calculating the trajectory execution deviation that incorporates environmental changes, model errors, and external disturbances. The online compensation algorithm then dynamically corrects the subsequent unexecuted joint command sequences based on this deviation. This mechanism expands the control closed loop from traditional, lagging feedback focused solely on the robot's own state to feedforward adjustment that includes external environment perception and model prediction. It can proactively compensate for anticipated deviations identified before execution but not yet occurring, enhancing the ability to suppress dynamic changes and system uncertainties, thereby achieving more stable and accurate task execution under complex operating conditions. Attached Figure Description

[0064] Figure 1 This is a flowchart illustrating the steps of the industrial robot 3D vision-guided AI precision control method described in this invention.

[0065] Figure 2 A flowchart for scene semantic segmentation;

[0066] Figure 3 Flowchart for building a dynamic environment model;

[0067] Figure 4 A multi-feature scatter plot for industrial robot trajectory analysis;

[0068] Figure 5 This is a graph comparing the commanded and actual values ​​of the joint angles of an industrial robot over time. Detailed Implementation

[0069] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0070] See Figure 1 The system acquires raw point cloud data from a 3D vision sensor and preprocesses it to generate scene understanding data. Based on the scene understanding data, a dynamic environment model describing the pose of the target object and its spatial relationship with the robot's end effector is constructed. Using the dynamic environment model and pre-defined robot task constraints, a spatial search algorithm generates multiple candidate robot motion trajectories to form a candidate trajectory set. The spatiotemporal features of each candidate robot motion trajectory in the candidate trajectory set are extracted and input into a pre-trained trajectory evaluation network to calculate and output a trajectory quality score for each candidate robot motion trajectory. Based on the trajectory quality score, the optimal robot motion trajectory is selected from the candidate trajectory set and decomposed into a sequence of joint space control commands that the robot can execute. The joint space control command sequence is sent to the robot controller to drive the robot to execute the motion, and the actual execution data is collected synchronously during the motion. The actual execution data is compared online with the predicted data of the dynamic environment model to generate trajectory execution deviation data. Based on the trajectory execution deviation data, the subsequent joint space control command sequence is corrected in real time through an online compensation algorithm to achieve closed-loop precise control.

[0071] See Figure 2 In one embodiment of the present invention, raw point cloud data containing the target object is acquired by a 3D vision sensor. The raw point cloud data is then preprocessed to obtain denoised 3D point cloud data. Subsequently, scene semantic segmentation is performed on the denoised 3D point cloud data to identify the structured 3D information of the target object and its surrounding environment to generate scene understanding data. Taking a gear-grabbing scene on an automotive parts assembly line as an example, the 3D vision sensor is fixed above the robot's workspace. The raw point cloud data it acquires includes the target object point cloud (i.e., the gear to be grasped), background point clouds such as the conveyor belt and frame, and noise point clouds generated by ambient light or reflections from metal surfaces. After receiving the raw point cloud data, statistical filtering is performed to remove spatially discrete outlier noise point clouds. This process involves calculating the average distance between each point and all points in its neighborhood, setting a distance threshold, and identifying and removing points with distances greater than the threshold. The point cloud data after statistical filtering is more spatially clustered. Voxel mesh downsampling is performed on the filtered point cloud data to reduce the point cloud density while preserving the scene's geometric features. Voxel mesh downsampling divides the 3D space into multiple tiny cubic grids, using the centroid coordinates of all points within each grid to approximate the points in that grid. The downsampling rate is set to 50%, reducing the original point cloud from one million points to five hundred thousand points. The formula for calculating the centroid coordinates is:

[0072]

[0073] in: This represents the coordinates of the centroid of the voxel mesh. This indicates the number of original point clouds that fall within this voxel grid. Indicates the first The three-dimensional coordinates of each point are obtained. Smoothing processing based on moving least squares is performed on the downsampled point cloud data to obtain denoised three-dimensional point cloud data with smooth surface and significantly reduced noise. The surface normal vector of the point cloud after smoothing is more continuous.

[0074] In some embodiments, the denoised 3D point cloud data is input into a pre-trained 3D semantic segmentation neural network model, which adopts the PointNet++ architecture. Through forward propagation computation of the 3D semantic segmentation neural network model, a semantic label is assigned to each point in the point cloud data. The semantic label set includes "target object," "carrying plane," "protective fence," and "irrelevant background." In the gear grasping scene, points on the gear surface are labeled as "target object," points on the conveyor belt surface are labeled as "carrying plane," points on the safety light curtain are labeled as "protective fence," and points on other distant frames are labeled as "irrelevant background." Based on the semantic labels, the point cloud data is clustered into different instance objects. A clustering algorithm based on Euclidean distance is used to aggregate points with the same semantic label and spatial proximity into a point cloud cluster, and the point cloud cluster of each instance object is extracted. The 3D bounding box, centroid coordinates, and surface normal vector of the point cloud cluster representing the target object are calculated. Simultaneously, the plane equation parameters of the carrying plane are extracted, collectively constituting structured 3D information describing the scene and generating scene understanding data. The plane equation parameters of the carrying plane are obtained by fitting using a random sampling consensus algorithm.

[0075] Optionally, the training data for the 3D semantic segmentation neural network model comes from manually labeled point clouds of a large number of similar industrial scenes. The training process minimizes the cross-entropy loss function between the predicted and true labels. It is understood that the order of point cloud preprocessing steps can be adjusted according to sensor characteristics, and the parameters for statistical filtering and voxel grid downsampling need to be set according to the actual point cloud density and noise level. In specific implementations, in the generated structured 3D information, the 3D bounding box of the target object uses an axial bounding box, the direction of which is determined by principal component analysis. The centroid coordinates are the arithmetic mean of the coordinates of all points in the point cloud cluster, and the surface normal vector is calculated by plane fitting of local neighborhood points. In some embodiments, the scene semantic segmentation step not only outputs semantic labels but also predicts the component label for each point within its instance. For example, for the "target object" gear, components such as teeth, holes, and end faces can be distinguished. Optionally, for environments with simple structures, the plane equation parameters of the bearing plane can be obtained by manually selecting point cloud regions and fitting them. It is understood that the process of generating scene understanding data is fully automated, requiring no manual intervention, and its output provides accurate and structured input for subsequent construction of dynamic environment models.

[0076] See Figure 3 In one embodiment of the present invention, a dynamic environment model describing the pose of a target object and its spatial relationship with the robot's end effector is constructed based on scene understanding data. The three-dimensional bounding box and centroid coordinates of the target object in the scene understanding data are analyzed to calculate the six-degree-of-freedom pose of the target object in the current robot base coordinate system. The pose of the robot's end effector in the initial state is calculated according to the robot kinematic model. The relative position and attitude relationship between the target object pose and the initial pose of the robot's end effector are calculated to construct an initial spatial relationship matrix. The target object pose, the initial spatial relationship matrix, and the geometric information of environmental obstacles are integrated with the three-dimensional map information of the robot's workspace to construct a dynamic environment model. In the gear grasping example scenario, a 3D bounding box representing the gear is parsed from the scene understanding data. Its center coordinates in the robot base coordinate system are (550.2 mm, 120.5 mm, 205.8 mm). The normalized three-axis direction vectors of the bounding box are (0.707, 0.0, 0.707), (0.0, 1.0, 0.0), and (-0.707, 0.0, 0.707). Based on these center coordinates and direction vectors, a rotation matrix and a translation vector are constructed to obtain the six-degree-of-freedom pose representation of the target object relative to the robot base coordinate system. The robot's kinematic model is a standard DH parameter model for a six-axis serial robot. Given that the angle values ​​of all joints are zero in the initial state, the pose of the robot's end effector is calculated using the forward kinematics chain product. Its position is (0.0 mm, 0.0 mm, 900.0 mm), and its orientation is the Euler angles (0.0 degrees, 0.0 degrees, 0.0 degrees) around the Z, Y, and X axes. The relative position and orientation relationship between the target object's pose and the robot's end effector's initial pose are calculated to construct an initial spatial relationship matrix. This matrix describes the transformation from the robot's end effector coordinate system to the target object's coordinate system, and its mathematical expression is as follows:

[0077]

[0078] in: This represents the pose transformation matrix of the target object relative to the robot's base coordinate system. This represents the pose transformation matrix of the robot's end effector relative to the robot's base coordinate system, representing the initial pose of the robot's end effector. This is the initial spatial relation matrix we are looking for. In the example, the translation part of this matrix is ​​approximately (550.2, 120.5, -694.2) millimeters.

[0079] In some embodiments, the 3D map information of the robot's workspace is obtained through prior offline scanning and modeling. This 3D map information includes triangular mesh models of fixed obstacles such as cabinets and supports. The integration process stores the calculated target object pose, initial spatial relationship matrix, and obstacle geometry information from the 3D map information together in the same data structure to form a dynamic environment model. The dynamic environment model exists in memory as a class object containing pose transformation matrices, bounding boxes, and mesh data. Optionally, the dynamic environment model update is triggered after the robot completes each motion segment or when new scene understanding data is transmitted from the 3D vision sensor. It is understood that the six-DOF pose calculation of the target object depends on the accuracy of the 3D bounding box in the scene understanding data; principal component analysis or iterative nearest-point algorithms can be used to optimize the bounding box orientation. In a specific implementation, the dynamic environment model is updated as the robot moves. After the robot begins to move towards the target object, the latest pose of the robot's end effector is calculated using the robot kinematics model based on the real-time joint angles fed back by the robot controller. This latest pose is then used to update the pose transformation matrix of the robot's end effector relative to the robot's base coordinate system in the dynamic environment model. In some embodiments, if the 3D vision sensor has real-time tracking capabilities, the pose of the target object in the dynamic environment model can also be corrected online based on the point cloud data of subsequent frames, thereby reflecting the possible minute movements of the target object. Optionally, the geometric information of environmental obstacles includes not only static obstacles but also temporary dynamic obstacle regions identified by additional sensors. It can be understood that the dynamic environment model is the global spatial reference basis for subsequent trajectory planning and online compensation, using the robot's base coordinate system as a unified reference framework. In specific implementations, the completed dynamic environment model, including the pose of the target object, the pose of the robot's end effector, and the geometric descriptions of all obstacles, is transformed and uniformly expressed in the robot's base coordinate system, ensuring the consistency of all spatial relationship calculations.

[0080] In one embodiment of the present invention, a set of candidate robot motion trajectories is formed by using a dynamic environment model combined with preset robot task constraints and a spatial search algorithm. The initial pose and target grasping pose of the robot end effector are obtained from the dynamic environment model. The initial pose is the pose of the robot end effector in its initial state, and the target grasping pose is the pose of the target object described in the dynamic environment model after being adjusted by a preset grasping offset. In the gear grasping example scenario, the initial pose comes from the initial pose of the robot end effector recorded in the dynamic environment model, and the target grasping pose is calculated based on the six-degree-of-freedom pose of the gear in the dynamic environment model combined with the preset grasping posture of the gripper. Random sampling is performed within the robot configuration space based on preset robot task constraints. These constraints include joint angular velocity limits, avoidance of singular configurations, and shortest path preference. During the random sampling process, random points are uniformly generated within the joint angle space of the six-axis robot. Each random point is a six-dimensional vector representing a robot configuration. Inverse kinematics verification is used during sampling to ensure that the end-effector pose corresponding to the configuration is reachable and meets joint constraints. Simultaneously, the condition number of the robot's Jacobian matrix under the configuration is calculated to reject configurations with a condition number greater than a threshold of 10, thereby avoiding singular configurations. Joint angular velocity limits are reflected by constraining the maximum joint angle difference between adjacent sampling points to 0.1 per radian. Based on sampling points, a path connection algorithm is used to plan a continuous path connecting the starting pose and the target grasping pose while avoiding obstacles in the dynamic environment model. The path connection algorithm adopts a bidirectional fast exploration random tree algorithm, growing two random trees from the starting point and the target point. In each iteration, a connection is attempted between the two trees. Collision detection of the connection edges is based on interference checks between the triangular mesh model of obstacles in the dynamic environment model and the cylindrical envelope model of the robot links until a collision-free continuous path is found. The path is represented by a series of sequentially connected joint space configuration points.

[0081] For each continuous path, time-parameterized interpolation is performed to generate a sequence of the end-effector's Cartesian space position and orientation over time, i.e., a candidate robot trajectory. The time-parameterized interpolation discretizes the path in joint space into one hundred uniformly distributed intermediate points, assigning a timestamp to each intermediate point to satisfy maximum joint velocity and acceleration constraints, generating a position sequence. Follow the formula:

[0082]

[0083] in: Represents the starting position vector. Represents the target position vector. This indicates that the normalized time parameter varies linearly from 0 to 1. It is a fifth-order polynomial scalar function used to achieve a smooth motion profile. Repeated planning and generation steps generate multiple candidate robot trajectories that differ in geometric paths or time parameters, forming a candidate trajectory set. By changing the seed value of the random number generator, a bidirectional fast-exploration random tree algorithm explores different expansion directions, thus generating different geometric paths. Different time parameterization functions are applied to the same geometric path. Variants such as third-order polynomials or sine functions can generate trajectories with different time distributions, ultimately forming a candidate trajectory set of ten candidate robot motion trajectories containing five different geometric paths, each path corresponding to two time profiles.

[0084] In some embodiments, the shortest path preference in robot task constraints is implemented in the path connection algorithm by introducing a target bias probability into the random tree expansion, extending directly towards the target placement point with a 20% probability to shorten the path length. In some embodiments, the collision detection algorithm not only checks for interference between the robot and static obstacles but also queries the dynamic environment model in real time to confirm that the target object has not moved during planning, thereby avoiding dynamic collisions. Optionally, the path connection algorithm can employ a probabilistic road graph algorithm, first randomly scattering points in the placement space to construct a connected graph and then searching for the shortest path. Optionally, time parameterized interpolation can be performed directly in Cartesian space, generating smooth Cartesian space trajectories by spline interpolation of the end-effector's position and orientation, and then converting them into joint space sequences through inverse kinematics. It is understood that the generated candidate robot motion trajectories have significant differences in their joint space sequences; some trajectories have large joint movement amplitudes while joint movement amplitudes are small, and some trajectories are the opposite. It is understood that the generation of the candidate trajectory set is an offline or online preprocessing step, and its computation time must meet the real-time requirements of the robot task.

[0085] In one embodiment of the present invention, the spatiotemporal features of each candidate robot motion trajectory in the candidate trajectory set are extracted and input into a pre-trained trajectory evaluation network to calculate and output the trajectory quality score of each candidate robot motion trajectory. Based on the trajectory quality score, the optimal robot motion trajectory is selected from the candidate trajectory set and decomposed into a sequence of joint space control instructions that the robot can execute. In the gear grasping example scenario, the candidate trajectory set contains ten candidate robot motion trajectories. For each candidate robot motion trajectory in the candidate trajectory set, its trajectory smoothness feature, minimum distance to obstacle feature, joint motion coordination feature, and expected execution time feature are extracted to form the spatiotemporal feature vector of the candidate robot motion trajectory. The trajectory smoothness feature is quantified by calculating the sum of squares of the second derivatives of the joint angle sequence, with a value range between 0.1 and 1.0. The minimum distance to obstacle feature is obtained by calculating the minimum Euclidean distance between the robot link envelope and the surfaces of all obstacles in the dynamic environment model, with a value between 5 mm and 50 mm. The joint motion coordination feature is measured by evaluating the standard deviation of the change in each joint angle, with a value range between 0.05 and 0.3 radians. The expected execution time feature is directly taken from the total time after time parameterization interpolation, with a value between 2.0 seconds and 5.0 seconds. The spatiotemporal feature vector corresponding to each candidate robot trajectory is sequentially input into a pre-trained trajectory evaluation network. This network is a multilayer perceptron structure with four neurons in the input layer corresponding to the four dimensions of the spatiotemporal feature vector, two fully connected layers with eight and four neurons respectively in the hidden layer, and a single neuron in the output layer. The trajectory evaluation network calculates a scalar value representing the overall quality of the trajectory through forward propagation, which is the trajectory quality score of the candidate robot trajectory. The forward propagation calculation involves linear transformations of the weight matrix and bias vector, as well as activation function processing. The calculation can be expressed as:

[0086]

[0087] in: This represents the spatiotemporal feature vector of the input. and This represents the weight matrix and bias vector of the first hidden layer. and This represents the weight vector and bias scalar from the second hidden layer to the output layer. Represents the ReLU activation function. The Sigmoid activation function compresses the output to between zero and one. The trajectory quality scores of all candidate robot trajectories in the candidate trajectory set are compared, and the candidate robot trajectory with the highest trajectory quality score is selected as the optimal robot trajectory. Using robot inverse kinematics, the end-effector pose at each time point on the optimal robot trajectory is calculated into the corresponding robot joint angle values. According to a preset control cycle, the continuous sequence of joint angle values ​​is discretized into a series of time-ordered joint angle target commands, forming a joint space control command sequence. See Table 1.

[0088] Table 1: Example Table of Spatiotemporal Features of Candidate Trajectory Sets and Trajectory Quality Scoring

[0089]

[0090] In some embodiments, the trajectory smoothness feature is calculated using the root mean square value of joint acceleration, and the joint motion coordination feature is measured using the variance of the ratio of the maximum velocity to the average velocity of each joint. In some embodiments, the training data for the trajectory evaluation network comes from the feature vectors of successful and failed trajectories in historical execution records and their manually labeled good and bad performance. The training process uses a mean squared error loss function and a stochastic gradient descent optimizer. Optionally, the spatiotemporal feature vector may include additional feature dimensions such as energy consumption estimation features or end-effector posture stability features. Optionally, a threshold filter may be introduced during the comparison of trajectory quality scores, considering only candidate robot motion trajectories with scores higher than a threshold of 0.8 for final selection. It is understood that robot inverse kinematics solutions may generate multiple sets of joint angle solutions for a single end-effector pose; in this case, the solution closest to the joint angle at the previous moment is selected to ensure motion continuity. It is understood that the discretized control cycle of the joint space control command sequence is usually consistent with the servo cycle of the robot controller, such as two milliseconds or four milliseconds. In the specific implementation, according to the data in Table 1, the trajectory quality score of trajectory number 7 is 0.95, which is the highest. Therefore, the candidate robot motion trajectory corresponding to trajectory number 7 is selected as the optimal robot motion trajectory. The optimal robot motion trajectory contains a sequence of end-effector poses at one hundred time points. For each pose, the robot inverse kinematics solver is called to obtain the corresponding six joint angle values, thus forming a continuous sequence containing one hundred sets of joint angle values. The sequence is resampled according to a control cycle of four milliseconds to generate two hundred and fifty joint angle target instructions arranged in chronological order, which constitute the joint space control instruction sequence.

[0091] See Figure 4This is a multi-feature scatter plot used for trajectory analysis of industrial robots. By combining color, size, and coordinates, it can quickly identify trajectories with optimal safety (distance), stability (coordination), and smoothness (such as trajectory 7), significantly reducing the time cost of manual screening. It can directly identify high-risk trajectories (such as trajectory 4) that are too close to obstacles or have poor joint coordination, avoiding robot collisions or vibration failures caused by selecting such trajectories. The visualized feature distribution can verify whether the output of the trajectory evaluation network conforms to engineering intuition, such as whether high-scoring trajectories show better multi-feature combinations in the graph, thereby improving the reliability of the algorithm. The analysis logic of this graph can be distilled into a standardized trajectory evaluation process and reused for robot trajectory planning in other industrial scenarios (such as welding and assembly), improving the decision-making efficiency of similar tasks.

[0092] In one embodiment of the present invention, a sequence of joint space control commands is sent to a robot controller to drive the robot to perform motion. During the motion, actual execution data is collected synchronously. The actual execution data is compared online with the predicted data from the dynamic environment model to generate trajectory execution deviation data. Based on the trajectory execution deviation data, an online compensation algorithm is used to correct subsequent joint space control command sequences in real time, achieving closed-loop precise control. In the gear grasping example scenario, the joint space control command sequence contains 250 joint angle target commands arranged in chronological order, with a control cycle of four milliseconds. The joint space control command sequence is sent one by one to the real-time control loop of the robot controller. The robot controller calculates the control voltage of each joint motor based on the received joint angle target commands and drives the robot's joint motors to move. Simultaneously, the robot's joint encoder reads the actual angle values ​​of each joint in real time with a sampling period of four milliseconds. The difference between the actual angle value sequence and the command value sequence of joint one fluctuates within a range of ±0.05 radians. A three-dimensional vision sensor collects the actual pose point cloud data of the robot's end effector and the target object in real time with a period of sixteen milliseconds. The actual angle values ​​and actual pose point cloud data are synchronously encapsulated with timestamps as the actual execution data output.

[0093] The actual execution data is compared online with the predicted data from the dynamic environment model to generate trajectory execution deviation data. The actual pose point cloud data of the robot end effector is parsed from the actual execution data, and its actual centroid pose is calculated. The actual centroid pose is obtained by calculating the average coordinates of all points in the point cloud cluster. At time t=0.5 seconds, the position components of the actual centroid pose of the robot end effector are (275.1 mm, 60.3 mm, 650.8 mm). The predicted pose of the robot end effector at the current time, i.e., t=0.5 seconds, is obtained from the dynamic environment model, and its position components are (275.0 mm, 60.0 mm, 651.0 mm). The differences between the actual centroid pose and the predicted pose in each of the position and attitude components are calculated. The position difference vector is (0.1 mm, 0.3 mm, -0.2 mm), and the attitude difference is expressed in Euler angles as (0.05 degrees, -0.1 degrees, 0.02 degrees). The position and attitude differences, as well as the differences between the actual joint angle values ​​in the actual execution data and the corresponding commands in the joint space control command sequence, are integrated into trajectory execution deviation data. The trajectory execution deviation data is a vector containing twelve components, with six components corresponding to the angle deviations of the six joints and six components corresponding to the position and attitude deviations of the end effector pose.

[0094] Based on trajectory execution deviation data, an online compensation algorithm corrects the subsequent joint space control command sequence in real time. The trajectory execution deviation data is input into the online compensation algorithm, which is built on the proportional-integral-differential principle. The online compensation algorithm calculates the compensation amount used to correct subsequent joint angle target commands based on the historical and current values ​​of the trajectory execution deviation data. The calculation follows the formula:

[0095]

[0096] in: This represents the trajectory execution deviation data vector at the current moment. , , These represent the pre-tuned proportional, integral, and differential coefficient matrices in the online compensation algorithm, respectively. This represents the integral of the historical trajectory deviation data from the initial moment to the current moment. This represents the rate of change of the trajectory execution deviation data at the current moment. The compensation amount is superimposed on subsequent joint angle target commands in the unexecuted joint space control command sequence to generate a corrected joint space control command sequence. Assuming the 120th command has been executed, the calculated joint angle compensation amount... A vector addition operation is performed with each joint angle target command in instructions 121 to 250. The corrected joint space control command sequence is then sent to the robot controller to replace the original command sequence to drive the robot to complete subsequent movements, forming a closed-loop precision control process of perception, decision-making, execution, and correction.

[0097] In some embodiments, the period for acquiring actual pose point cloud data by the 3D vision sensor is not synchronized with the robot control cycle. An interpolation algorithm is used to align the timestamps of the actual pose point cloud data to the control cycle time. In some embodiments, the pose difference in the trajectory execution deviation data is calculated using quaternion representation to obtain a smoother pose error metric. Optionally, an anti-saturation limit is set for the integral term in the online compensation algorithm to prevent the integral term from becoming too large due to long-term deviation accumulation. Optionally, the acquisition of actual execution data and the calculation of trajectory execution deviation data are completed in a separate real-time thread to ensure the timeliness of online compensation. It is understood that the online compensation algorithm operates on the subsequent part of the joint space control command sequence and does not produce retrospective corrections for commands that have been sent or are being executed.

[0098] See Figure 5 This is a graph comparing the commanded and actual values ​​of an industrial robot's joint angles over time, which has significant engineering value in precise robot control scenarios. As can be seen from the graph, the fluctuation amplitude did not diverge significantly throughout the entire movement, indicating that the control system was in a stable state with no risk of loss of control. The error amplitude remained within a small range, indicating that the robot's joint control accuracy was high and could meet the needs of most industrial scenarios. This graph visually verifies the effectiveness of the robot's closed-loop joint control; the actual angle value closely follows the commanded value, which is core evidence of the system's stable operation. The amplitude and frequency of high-frequency fluctuations provide a quantitative basis for controller parameter optimization; for example, oscillations can be suppressed by adjusting PID parameters to further improve tracking accuracy. For AI precision control algorithms, this graph can be used to verify the effect of online compensation algorithms, compare the error changes before and after compensation, and evaluate the actual gain of the algorithm.

[0099] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A method for precise AI control of industrial robots guided by 3D vision, characterized in that, include: Acquire raw point cloud data collected by a 3D vision sensor and preprocess it to generate scene understanding data; Based on scene understanding data, a dynamic environment model is constructed to describe the pose of the target object and its spatial relationship with the robot's end effector. Using a dynamic environment model and combining it with preset robot task constraints, multiple candidate robot motion trajectories are generated through a spatial search algorithm, forming a candidate trajectory set; The spatiotemporal features of each candidate robot motion trajectory in the candidate trajectory set are extracted, input into a pre-trained trajectory evaluation network for calculation, and the trajectory quality score of each candidate robot motion trajectory is output. Based on the trajectory quality score, the optimal robot motion trajectory is selected from the candidate trajectory set and decomposed into a sequence of joint space control instructions that the robot can execute; The joint space control command sequence is sent to the robot controller to drive the robot to perform motion, and the actual execution data is collected synchronously during the motion. The actual execution data is compared online with the predicted data from the dynamic environment model to generate trajectory execution deviation data; Based on trajectory execution deviation data, the subsequent joint space control command sequence is corrected in real time through an online compensation algorithm to achieve closed-loop precise control.

2. The industrial robot three-dimensional vision-guided AI precision control method according to claim 1, characterized in that, Acquire raw point cloud data from a 3D vision sensor and preprocess it to generate scene understanding data, including: The system acquires raw point cloud data containing the target object collected by a 3D vision sensor, and preprocesses the raw point cloud data to obtain denoised 3D point cloud data. Scene semantic segmentation is performed on the denoised 3D point cloud data to identify the structured 3D information of the target object and its surrounding environment, and to generate scene understanding data. The process of acquiring raw point cloud data containing the target object collected by a 3D vision sensor and preprocessing the raw point cloud data to obtain denoised 3D point cloud data specifically involves: Receive raw point cloud data collected by a 3D vision sensor in the robot's workspace, wherein the raw point cloud data includes target object point cloud, background point cloud and noise point cloud; Perform statistical filtering on the original point cloud data to remove spatially discrete outlier noise point clouds; Voxel mesh downsampling is performed on the filtered point cloud data to reduce the point cloud density while preserving the geometric features of the scene; Smoothing is performed on the downsampled point cloud data to obtain denoised 3D point cloud data with smooth surface and significantly reduced noise.

3. The industrial robot three-dimensional vision-guided AI precision control method according to claim 2, characterized in that, Scene semantic segmentation is performed on the denoised 3D point cloud data to identify the structured 3D information of the target object and its surrounding environment, generating scene understanding data, specifically: The denoised 3D point cloud data is input into the 3D semantic segmentation neural network model; A semantic label is assigned to each point in the point cloud data using a 3D semantic segmentation neural network model. The semantic label includes at least the target object, the bearing plane, the protective fence, and the irrelevant background. The point cloud data is clustered into different instance objects based on semantic tags, and the point cloud clusters of each instance object are extracted. The system calculates the 3D bounding box, centroid coordinates, and surface normal vectors of the point cloud clusters representing the target object, and extracts the plane equation parameters of the bearing plane. Together, they constitute structured 3D information describing the scene and generate scene understanding data.

4. The industrial robot three-dimensional vision-guided AI precision control method according to claim 3, characterized in that, Based on scene understanding data, a dynamic environment model describing the pose of the target object and its spatial relationship with the robot's end effector is constructed, specifically as follows: Analyze the scene to understand the 3D bounding box and centroid coordinates of the target object in the data, and calculate the six-degree-of-freedom pose of the target object in the current robot base coordinate system; Calculate the pose of the robot's end effector in the initial state based on the robot's kinematic model; Calculate the relative position and orientation relationship between the target object pose and the initial pose of the robot end effector, and construct the initial spatial relationship matrix; By combining the 3D map information of the robot's workspace, the target object's pose, the initial spatial relationship matrix, and the geometric information of environmental obstacles are integrated to construct a dynamic environment model, which is updated as the robot moves.

5. The industrial robot three-dimensional vision-guided AI precision control method according to claim 4, characterized in that, Using a dynamic environment model and pre-defined robot task constraints, a spatial search algorithm generates multiple candidate robot motion trajectories, forming a candidate trajectory set, specifically: Obtain the initial pose of the robot's end effector and the target grasping pose from the dynamic environment model; Based on preset robot task constraints, including joint angular velocity limits, avoidance of singular configurations, and shortest path preference, random sampling is performed within the robot configuration space. Based on sampling points, a path connection algorithm is used to plan a continuous path connecting the starting pose and the target grasping pose while avoiding obstacles in the dynamic environment model. For each continuous path, time parameter interpolation is performed to generate a sequence of the end-effector's Cartesian spatial position and orientation as a function of time, which is a candidate robot trajectory. Repeat the planning and generation steps to generate multiple candidate robot motion trajectories that differ in geometric path or time parameters, forming a candidate trajectory set.

6. The industrial robot three-dimensional vision-guided AI precision control method according to claim 5, characterized in that, The spatiotemporal features of each candidate robot trajectory in the candidate trajectory set are extracted, input into a pre-trained trajectory evaluation network for calculation, and the trajectory quality score of each candidate robot trajectory is output, specifically: For each candidate robot motion trajectory in the candidate trajectory set, extract its trajectory smoothness features, minimum distance to obstacle features, joint motion coordination features, and expected execution time features to form the spatiotemporal feature vector of the candidate robot motion trajectory. The spatiotemporal feature vectors corresponding to each candidate robot motion trajectory are sequentially input into a pre-trained trajectory evaluation network; The trajectory evaluation network is a multilayer perceptron structure. It outputs a scalar value that represents the overall quality of the trajectory through forward propagation calculation, which is the trajectory quality score of the candidate robot's motion trajectory.

7. The industrial robot three-dimensional vision-guided AI precision control method according to claim 6, characterized in that, Based on the trajectory quality score, the optimal robot motion trajectory is selected from the candidate trajectory set and decomposed into a sequence of joint space control commands that the robot can execute, specifically: Compare the trajectory quality scores of all candidate robot motion trajectories in the candidate trajectory set, and select the candidate robot motion trajectory with the highest trajectory quality score as the optimal robot motion trajectory; By using robot inverse kinematics, the end-effector pose at each time point on the optimal robot motion trajectory is calculated into the corresponding robot joint angle values. According to the preset control cycle, the continuous sequence of joint angle values ​​is discretized into a series of joint angle target commands arranged in chronological order, forming a joint space control command sequence.

8. The industrial robot three-dimensional vision-guided AI precision control method according to claim 7, characterized in that, The joint space control command sequence is sent to the robot controller to drive the robot to perform movements, and the actual execution data is collected synchronously during the movements. Specifically: The joint space control command sequence is sent one by one to the real-time control loop of the robot controller; The robot controller drives the motors of each joint of the robot to move according to the joint space control command sequence; Meanwhile, the actual angle values ​​of each joint are read in real time through the joint encoder of the robot body, and the actual pose point cloud data of the robot end tool and target object are collected in real time through the three-dimensional vision sensor. The actual angle value and the actual pose point cloud data are synchronously encapsulated and output as the actual execution data.

9. The industrial robot three-dimensional vision-guided AI precision control method according to claim 8, characterized in that, The actual execution data is compared online with the predicted data from the dynamic environment model to generate trajectory execution deviation data, specifically: The actual pose point cloud data of the robot's end effector is extracted from the actual execution data, and its actual centroid pose is calculated. Obtain the predicted pose of the robot's end effector at the current moment from the dynamic environment model; Calculate the difference between the actual centroid pose and the predicted pose in each of the position and attitude components; The difference, along with the difference between the actual joint angle value in the actual execution data and the corresponding instruction in the joint space control instruction sequence, are integrated into trajectory execution deviation data.

10. The industrial robot three-dimensional vision-guided AI precision control method according to claim 9, characterized in that, Based on trajectory execution deviation data, the subsequent joint space control command sequence is corrected in real time through an online compensation algorithm to achieve closed-loop precise control, specifically: The trajectory execution deviation data is input into the online compensation algorithm, which is constructed based on the proportional-integral-differential principle. The online compensation algorithm calculates the compensation amount used to correct subsequent joint angle target commands based on the historical and current values ​​of trajectory execution deviation data. The compensation amount is superimposed on the subsequent joint angle target commands in the joint space control command sequence that has not yet been executed, to generate the corrected joint space control command sequence. The corrected joint space control command sequence is sent to the robot controller to replace the original command sequence, so as to drive the robot to complete subsequent movements and form a closed-loop precision control process of perception, decision-making, execution and correction.

Images