Workpiece grabbing method and system based on visual feedback

Abstract

The invention relates to the technical field of automatic workpiece grabbing and visual servo control, and discloses a workpiece grabbing method and system based on visual feedback, and the method comprises the steps: obtaining a real-time image through a front-view camera, a side-view camera and a top-view camera, and extracting surface features through a feature fusion network; inputting the features into a pose estimation model based on a particle filtering framework, and iteratively updating the pose in combination with a multi-dimensional observation likelihood function to generate prediction parameters; constructing a multi-target path planning model based on the parameters, and optimizing the path by adopting a dynamic planning algorithm with the shortest path and the minimum joint movement as targets; a hierarchical control model is established, a strategy layer performs global planning, an adjustment layer corrects a local path, and an execution layer realizes trajectory tracking through visual servo control and outputs a control instruction to complete grabbing. Through multi-view perception, robust pose estimation, global optimization path planning and hierarchical control, the precision, efficiency and stability of workpiece grabbing in a complex environment are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic workpiece grasping and visual servo control, and in particular to a workpiece grasping method and system based on visual feedback. Background Art

[0002] In scenarios such as industrial automated production, logistics sorting, and intelligent manufacturing, workpiece grasping is a key link in achieving automated processes, and its efficiency and precision directly affect overall production performance. Traditional workpiece grasping methods often rely on preset programs or fixed trajectory control, which makes it difficult to adapt to dynamic scenarios with variable workpiece postures and complex environments. For example, on an assembly line, workpieces may shift in position due to conveying vibrations, or the grasping point may be difficult to identify due to stacking or occlusion. Traditional robotic arms often lack real-time perception and adjustment capabilities, resulting in grasping failures and workpiece damage.

[0003] With the development of machine vision technology, grasping methods based on visual feedback have gradually become a research hotspot. However, existing technologies still have many limitations: First, the image information obtained by single-view vision devices is limited, making it difficult to fully restore the three-dimensional shape of the workpiece. This is especially true when the workpiece surface has complex textures or occlusions, which can easily lead to incomplete feature extraction and affect the accuracy of pose estimation. Second, pose estimation models often use a single feature matching method, such as relying solely on edge contours or key feature points, ignoring multi-dimensional information such as texture and occlusion, resulting in insufficient robustness of pose prediction in dynamic environments. Third, path planning often optimizes a single goal (such as shortest path) without considering factors such as the range of motion of the robot arm's joints and energy consumption. This can easily lead to joint motion overruns or path jitter, reducing grasping stability. Fourth, control models are mostly single-level structures, making it difficult to balance global planning and local corrections. In the event of sudden obstacles or path deviations, the trajectory cannot be quickly adjusted, resulting in reduced tracking accuracy.

[0004] In existing technologies, multi-view feature fusion often suffers from insufficient correction of viewpoint bias, resulting in deviations between the fused features and the actual workpiece morphology. Particle filtering frameworks fail to effectively incorporate multi-dimensional observation constraints during particle degradation, resulting in slow convergence of pose estimation. Dynamic programming algorithms lack an adaptive cost adjustment mechanism during state transitions, leading to inefficient path search. Hierarchical control suffers from a high degree of coupling between the strategy layer and the execution layer, making rapid response difficult. These issues collectively restrict the application of visual feedback grasping technology in high-precision, high-dynamic scenarios. A comprehensive solution integrating multi-view perception, robust pose estimation, globally optimized path planning, and hierarchical control is needed. Summary of the Invention

[0005] The purpose of the present invention is to provide a workpiece grasping method and system based on visual feedback to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a workpiece grasping method based on visual feedback, the method comprising: Real-time image data of a target workpiece is acquired through a multi-view visual perception device, the multi-view visual perception device including a front-view camera, a side-view camera, and a top-view camera; feature extraction processing is performed on the real-time image data based on a feature fusion network to obtain workpiece surface feature data; the workpiece surface feature data is input into a pre-trained pose estimation model, and the pose estimation model uses a particle filter framework to iteratively update the workpiece pose based on an observation likelihood function to generate workpiece pose prediction parameters; A multi-objective path planning model is constructed based on the workpiece posture prediction parameters, wherein the multi-objective path planning model takes the shortest grasping path length and the smallest joint motion amplitude as optimization goals, and globally optimizes the grasping path using a dynamic programming algorithm, wherein the dynamic programming algorithm introduces a state transition cost and a heuristic evaluation function; and optimal grasping path data is output based on the multi-objective path planning model; A hierarchical grasping control model is established based on the optimal grasping path data, and the hierarchical grasping control model includes a strategy layer, an adjustment layer and an execution layer, wherein the strategy layer performs global grasping planning based on the workpiece posture prediction parameters, the adjustment layer performs local path correction based on the optimal grasping path data, and the execution layer realizes the trajectory tracking of the robot arm based on the visual servo control algorithm; the hierarchical grasping control model outputs grasping control instructions to realize visual feedback grasping control of the target workpiece.

[0007] Preferably, the workpiece surface feature data is input into a pre-trained pose estimation model, the pose estimation model adopts a particle filter framework, and iteratively updates the workpiece pose based on the observation likelihood function to generate workpiece pose prediction parameters, including: Acquire real-time image data, the real-time image data including workpiece edge contour information, surface texture information, key feature point coordinates, and background environmental occlusion information; construct a state space based on the real-time image data, and construct an action space based on the translational motion, rotational angle, and gripper opening and closing degree executable by the end effector of the robotic arm; Constructing a multidimensional observation likelihood function based on the state space and the action space, the multidimensional observation likelihood function including a contour matching term, a texture alignment term, a feature point coincidence term, and an occlusion exclusion term, wherein the contour matching term is calculated by the shape similarity between the actual contour of the workpiece and the predicted contour, the texture alignment term is calculated by the grayscale correlation between the workpiece surface texture and the template texture, the feature point coincidence term is calculated by the position deviation between the actual feature point coordinates and the predicted feature point coordinates, and the occlusion exclusion term is calculated by the pixel separation between the background area and the workpiece area; Constructing a particle filter framework, the particle filter framework includes an initial particle set, a state transfer function, and an observation likelihood calculation module. The initial particle set contains multiple workpiece pose hypothesis samples. The state transfer function predicts and updates the particle poses using a robotic arm kinematic model. The observation likelihood calculation module evaluates particle weights based on the multidimensional observation likelihood function. The initial particle set is updated by a resampling method. High-weight particles are retained through roulette wheel selection, low-weight particles are eliminated, and new particles are added. The mean and covariance of the workpiece pose are calculated based on the updated particle set. The pose prediction parameters are output based on the particle filter framework. The pose prediction parameters include particle mean coordinates, covariance matrix, key feature point matching confidence, and occlusion area exclusion probability.

[0008] Preferably, a multi-objective path planning model is constructed based on the workpiece posture prediction parameters. The multi-objective path planning model takes the shortest grasping path length and the smallest joint motion amplitude as optimization goals, and adopts a dynamic programming algorithm to perform global optimization on the grasping path. Outputting the optimal grasping path data based on the multi-objective path planning model includes: Constructing a multi-objective function for path planning, the multi-objective function including a path length optimization objective function and a joint motion optimization objective function, wherein the path length optimization objective function is calculated by summing the ratios of the straight-line distances of the path segments to the corresponding movement speeds, and the joint motion optimization objective function is calculated by weighted summing of the rotation angles, translational motions, and acceleration changes of each joint; Constructing path planning constraints based on the multi-objective function, the path planning constraints include speed constraints, angle constraints, acceleration constraints, and obstacle constraints. The speed constraints are used to limit the range of movement speed of the end effector of the manipulator, the angle constraints are used to limit the range of variation of the rotation angles of each joint, the acceleration constraints are used to limit the range of values ​​of the manipulator's motion acceleration, and the obstacle constraints are used to ensure a safe distance between the manipulator and surrounding fixed obstacles. The grasping path is encoded using a node sequence, where each node contains spatial coordinates and robot arm joint angle information. A state transition probability is constructed based on the state transition cost and a heuristic evaluation value, and the next feasible node is determined using the state transition probability. The heuristic evaluation value is calculated by weighting the straight-line distance between nodes and the joint motion amplitude. Introducing an adaptive state transfer cost mechanism, the adaptive state transfer cost mechanism adopts a time-varying weight coefficient, the time-varying weight coefficient changes linearly with the increase of the number of planning iterations, and the state transfer cost is dynamically adjusted by the time-varying weight coefficient; A dynamic heuristic evaluation mechanism is introduced. The dynamic heuristic evaluation mechanism uses a time-varying influence factor that increases linearly with the number of planning iterations. The time-varying influence factor is used to adjust the weight of the heuristic evaluation value in the state transition probability. Iterative optimization is performed based on the adaptive state transition cost mechanism and the dynamic heuristic evaluation mechanism. The length cost and joint motion cost of the path generated by each iteration are evaluated, and non-dominated solutions are added to the Pareto front solution set. An optimal solution that satisfies the length-motion trade-off is selected from the Pareto front solution set as the optimal path, and a spline curve smoothing process is performed on the optimal path to generate an optimal node sequence and corresponding robot arm motion parameters.

[0009] Preferably, the strategy layer performs global grasping planning based on the workpiece pose prediction parameters, including: A parameterized curve is used to describe the grasping path, and the grasping path is expressed as a function of a path parameter, wherein the value of the path parameter ranges from 0 to 1, and the grasping path includes an X-axis coordinate component, a Y-axis coordinate component, and a Z-axis coordinate component; The grasping path is described based on a quintic Bezier curve, and the spatial coordinates of the grasping path are calculated by summing the products of the control vertex coordinates and the Bezier basis functions, wherein the Bezier basis functions are calculated by the power function of the combination number and the path parameter; Constructing grasping path constraints, the grasping path constraints including position constraints, velocity constraints, acceleration constraints, and curvature constraints. The position constraint is used to limit the coordinate range of the grasping path, the velocity constraint is used to limit the first-order derivative range of the grasping path, the acceleration constraint is used to limit the second-order derivative range of the grasping path, and the curvature constraint is used to limit the curvature variation range of the grasping path. Constructing a global multi-objective optimization function, the global multi-objective optimization function including a total path length term, a curvature integral term, a curvature change rate integral term, and an obstacle distance measurement term, and performing a weighted combination of each term in the global multi-objective optimization function using a weighting coefficient; Discretize the grasping path interval into multiple path segments, discretize the global multi-objective optimization function, and construct a global discretization objective function, wherein the global discretization objective function includes path segment length, path segment curvature, curvature change, and minimum obstacle distance; Iteratively optimizing the global discretized objective function using a gradient descent method, and updating the position coordinates of the control vertex along the negative gradient direction by calculating the gradient value of the objective function with respect to the control vertex; The optimized grasping path is smoothed by cubic spline interpolation. By maintaining the continuity of the position derivative, velocity derivative, and acceleration derivative at the interpolation endpoints, a smooth and continuous grasping path is generated. A node sequence and the corresponding robot arm movement speed are generated based on the smooth and continuous grasping path.

[0010] Preferably, the adjustment layer performs local path correction based on the optimal crawling path data, including: A local planning window is constructed based on the current position, current joint angle, and current movement speed of the robot arm, and the size of the local planning window is adaptively adjusted using a speed correlation coefficient to establish a positive correlation between the size of the local planning window and the current movement speed; A local environment model is constructed using multi-view visual data, and the image data is transformed to obtain position data in the local coordinate system. The occupancy probability of the grid map is updated based on the position data, and the occupancy probability value of each grid is calculated using the probability accumulation method. The Kalman filter algorithm is used to track dynamic obstacles. The state vector of the obstacle is predicted through the state prediction equation. The predicted state is updated based on the measurement data to obtain the precise position and motion information of the obstacle. Constructing a path correction model, using the kinematic equation of the manipulator as a state equation, the state equation including spatial coordinates, movement speed, and joint angles, and constructing state constraints and dynamic constraints. The state constraints are used to limit the value ranges of the spatial coordinates and the movement speed, and the dynamic constraints are used to limit the value ranges of the angular velocity, angular acceleration, and the rate of change of the joint angles; Constructing a multi-objective cost function, the multi-objective cost function including a reference trajectory tracking term, an obstacle avoidance term, a trajectory smoothing term, and an energy consumption term, and performing a weighted combination of each term in the multi-objective cost function using a weighting coefficient; The Lagrange multiplier method is used to optimize and solve the multi-objective cost function, a Lagrange function is constructed and constraints are introduced, and the optimal control quantity is obtained by solving the partial derivative equation group.

[0011] Preferably, the execution layer implements trajectory tracking of the robot arm based on a visual servo control algorithm, outputs grasping control instructions through the hierarchical grasping control model, and implements visual feedback grasping control of the target workpiece, including: Establishing a six-degree-of-freedom kinematic model for the robotic arm, the six-degree-of-freedom kinematic model includes a translation equation and a rotation equation, the translation equation includes a driving force term, a friction term, and a gravity term, and the rotation equation includes a torque term and a moment of inertia term; Constructing the six-degree-of-freedom kinematic model into a state-space expression, wherein the state vector of the state-space expression includes spatial coordinates, movement speed, joint angles, and angular velocity, and the control vector of the state-space expression includes driving force and three-axis torque; Linearizing the state space expression, calculating the partial derivatives of the system state equation with respect to the state vector and the control vector, and constructing a linearized prediction model; Constructing a trajectory tracking prediction cost function, wherein the trajectory tracking prediction cost function includes a tracking error term, a control amount penalty term, and a control increment penalty term, and performing a weighted combination of each penalty term through a weight matrix; Constructing state constraints, including space coordinate constraints and joint angle constraints, constructing control constraints, including driving force constraints and torque constraints, and constructing control increment constraints, including driving force increment constraints and torque increment constraints; Converting the trajectory tracking prediction cost function into a standard form of a quadratic programming problem, calculating the quadratic matrix and the linear term coefficients, and constructing the inequality constraint matrix and the equality constraint matrix; The effective set method is used to solve the quadratic programming problem, and the optimal solution is gradually approached by identifying effective constraints and solving subproblems; Based on the optimization solution, control quantity mapping is performed, the total driving force is distributed to each driving motor through the driving force distribution matrix, and the three-axis torque is distributed to each joint driver through the torque distribution matrix; The output of the drive motor is limited, the output torque is limited according to the rated power of the motor, and the rotation angle is limited according to the maximum rotation speed of the joint to generate the final grasping control instruction.

[0012] Preferably, feature extraction processing is performed on the real-time image data based on a feature fusion network to obtain workpiece surface feature data, including: grayscale processing is performed on the real-time image data obtained by the front-view camera, the side-view camera and the top-view camera respectively to obtain single-channel grayscale images; Gaussian filtering and denoising are performed on each grayscale image, and the local mean and variance are calculated through a sliding window to adjust the image contrast; a scale-invariant feature transformation algorithm is used to extract local feature points of each image, and a correspondence between images of different perspectives is established through feature point matching; the coordinates of the matched feature points are converted to a unified world coordinate system, and the multi-perspective feature point information is fused through a weighted average method to generate workpiece surface feature data containing depth information.

[0013] Preferably, constructing a particle filter framework includes: initializing a particle set using a uniform distribution, where each particle contains the position coordinates and rotation angle of the workpiece in three-dimensional space; calculating the end effector pose corresponding to the particle through the forward solution of the robot arm kinematics, projecting the particle pose to the image plane based on the camera extrinsic parameter matrix, and generating predicted feature point coordinates; calculating the pixel error between the predicted feature point coordinates and the actual image feature point coordinates, and constructing an observation likelihood function based on the sum of squared errors; obtaining the weight value of each particle through normalization processing, retaining particles with weight values ​​greater than a preset threshold, and eliminating particles with weight values ​​less than the preset threshold.

[0014] Preferably, constructing the state transition cost includes: calculating the Euclidean distance between the current node and the next node as the path length cost; calculating the change in rotation angle of each joint required for the robot arm to move from the current node to the next node as the joint motion cost; calculating the minimum distance to the fixed obstacle in the path, and increasing the penalty cost if it is less than the safety distance; generating the total state transition cost by linearly combining the path length cost, joint motion cost and penalty cost.

[0015] Preferably, the present invention further includes a workpiece grasping system based on visual feedback, for implementing the above-mentioned workpiece grasping method based on visual feedback, the system comprising: The first module is configured to acquire real-time image data of a target workpiece through a multi-view visual perception device, the multi-view visual perception device including a front-view camera, a side-view camera, and a top-view camera; perform feature extraction processing on the real-time image data based on a feature fusion network to obtain workpiece surface feature data; input the workpiece surface feature data into a pre-trained pose estimation model, which uses a particle filter framework to iteratively update the workpiece pose based on an observation likelihood function to generate workpiece pose prediction parameters; The second module is used to construct a multi-objective path planning model based on the workpiece posture prediction parameters. The multi-objective path planning model takes the shortest grasping path length and the smallest joint motion amplitude as optimization goals, and uses a dynamic programming algorithm to globally optimize the grasping path. The dynamic programming algorithm introduces state transition cost and heuristic evaluation function to improve path search efficiency; and outputs optimal grasping path data based on the multi-objective path planning model; The third module is used to establish a hierarchical grasping control model based on the optimal grasping path data. The hierarchical grasping control model includes a strategy layer, an adjustment layer and an execution layer. The strategy layer performs global grasping planning based on the workpiece posture prediction parameters, the adjustment layer performs local path correction based on the optimal grasping path data, and the execution layer realizes the trajectory tracking of the robot arm based on the visual servo control algorithm; the hierarchical grasping control model outputs grasping control instructions to realize visual feedback grasping control of the target workpiece.

[0016] Compared with the prior art, the present invention has the following beneficial effects: The system utilizes multi-perspective visual perception devices for front, side, and top views, enabling comprehensive acquisition of 3D workpiece image data. Combined with a feature fusion network, it extracts and fuses multi-perspective features, effectively overcoming the incomplete information inherent in single-perspective images. Through grayscale conversion, Gaussian filtering, and scale-invariant feature transformation, it achieves precise matching of feature points from different perspectives and unified coordinate system conversion. The resulting workpiece surface feature data includes complete edge contours, surface textures, key feature points, and depth information, providing a rich input basis for subsequent pose estimation and significantly improving the integrity and accuracy of feature extraction in complex environments.

[0017] The pose estimation model employs a particle filter framework and introduces a multi-dimensional observation likelihood function. Through a comprehensive evaluation of contour matching, texture alignment, feature point overlap, and occlusion exclusion, it achieves dynamic, iterative updates of the workpiece pose. Compared to traditional single-feature matching methods, this model effectively integrates multi-source information, resisting interference from background occlusion and illumination changes. Furthermore, a resampling mechanism retains high-weighted particles, improving the robustness and convergence speed of pose prediction. This ensures that predicted parameters (such as particle mean coordinates and covariance matrix) more closely match the actual workpiece state, providing accurate foundational data for path planning.

[0018] The multi-objective path planning model optimizes path length and joint motion by minimizing the length of the shortest path. It combines a dynamic programming algorithm with state transition costs and a heuristic evaluation function. This model achieves efficient global path optimization through adaptive time-varying weight coefficients and influencing factors. Compared to single-objective planning, this model ensures path efficiency while avoiding over-limit motion of the manipulator's joints, reducing joint wear and energy consumption. Furthermore, by selecting the optimal path through the Pareto front solution set and applying spline smoothing, the path better matches the manipulator's kinematic characteristics, enhancing the stability of the grasping process.

[0019] The hierarchical grasping control model achieves an organic combination of global planning and local correction through the collaborative work of the strategy layer, adjustment layer, and execution layer. The strategy layer performs global multi-objective optimization based on posture parameters to ensure the overall rationality of the path; the adjustment layer can quickly respond to sudden obstacles and path deviations through local window adaptive adjustment and Kalman filter tracking; the execution layer achieves high-precision trajectory tracking based on visual servo control and quadratic programming. This hierarchical structure reduces the coupling between modules and improves the system's adaptability to dynamic environments, allowing the robot arm to maintain high grasping accuracy and efficiency in complex scenarios, effectively meeting the high-precision and high-reliability requirements for workpiece grasping in industrial automation. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1This is a working principle diagram of the workpiece grasping method based on visual feedback according to the present invention; Figure 2 Flowchart for iterative updates of the pose estimation model; Figure 3 Flowchart for optimization of multi-objective path planning model; Figure 4 A flowchart for global capture planning at the strategy level; Figure 5 Flowchart for local path correction of adjustment layers. DETAILED DESCRIPTION

[0021] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0022] See also Figure 1-Figure 5 The present invention relates to a workpiece grasping method based on visual feedback, and the specific implementation steps are as follows: Real-time image data of a target workpiece is acquired through a multi-view visual perception device, the multi-view visual perception device including a front-view camera, a side-view camera, and a top-view camera; feature extraction processing is performed on the real-time image data based on a feature fusion network to obtain workpiece surface feature data; the workpiece surface feature data is input into a pre-trained pose estimation model, and the pose estimation model uses a particle filter framework to iteratively update the workpiece pose based on an observation likelihood function to generate workpiece pose prediction parameters; A multi-objective path planning model is constructed based on the workpiece posture prediction parameters, wherein the multi-objective path planning model takes the shortest grasping path length and the smallest joint motion amplitude as optimization goals, and globally optimizes the grasping path using a dynamic programming algorithm, wherein the dynamic programming algorithm introduces a state transition cost and a heuristic evaluation function; and optimal grasping path data is output based on the multi-objective path planning model; A hierarchical grasping control model is established based on the optimal grasping path data, and the hierarchical grasping control model includes a strategy layer, an adjustment layer and an execution layer, wherein the strategy layer performs global grasping planning based on the workpiece posture prediction parameters, the adjustment layer performs local path correction based on the optimal grasping path data, and the execution layer realizes the trajectory tracking of the robot arm based on the visual servo control algorithm; the hierarchical grasping control model outputs grasping control instructions to realize visual feedback grasping control of the target workpiece.

[0023] Example 1: When feeding workpiece surface feature data into a pre-trained pose estimation model, real-time image data is required. This real-time image data contains multiple aspects of information, including workpiece edge contour information, which outlines the workpiece's external shape; surface texture information, which reflects the grain characteristics of the workpiece's surface; coordinates of key feature points, which are important for determining the workpiece's position and posture; and background occlusion information, which is used to understand the obstruction of the workpiece by its surroundings. Based on this real-time image data, a state space is constructed, and at the same time, an action space is constructed based on the translational motion, rotational angle, and gripper opening and closing degree that the robot arm's end effector can perform.

[0024] Based on the constructed state space and action space, a multi-dimensional observation likelihood function is constructed, which contains several important terms. Among them, the contour matching term is obtained by calculating the shape similarity between the actual contour of the workpiece and the predicted contour. The calculation of shape similarity allows the model to better judge the degree of fit between the predicted contour and the actual contour. The texture alignment term is obtained by calculating the grayscale correlation between the workpiece surface texture and the template texture. The calculation of grayscale correlation can reflect the similarity between the textures. The feature point coincidence term is obtained by calculating the positional deviation between the actual feature point coordinates and the predicted feature point coordinates. The size of the positional deviation reflects the accuracy of the feature point prediction. The occlusion exclusion term is obtained by calculating the pixel separation between the background area and the workpiece area. The pixel separation helps the model eliminate the influence of background occlusion.

[0025] A particle filter framework is constructed, consisting of multiple components. The initial particle set contains multiple hypotheses about the workpiece's poses, representing various hypotheses about possible poses. The state transfer function uses the robot's kinematic model to predict and update the particle poses. This model predicts the particle's pose at the next moment based on the current state. The observation likelihood calculation module evaluates particle weights based on the previously constructed multidimensional observation likelihood function. This evaluation determines which particles are closest to the workpiece's true pose.

[0026] The initial particle set is updated using a resampling method. A roulette wheel selection process is used to retain high-weight particles, which are more likely to represent the workpiece's true pose. Low-weight particles are removed and new particles are added. This continuously optimizes the particle set to better reflect the workpiece's actual position. The mean and covariance of the workpiece's pose are calculated based on the updated particle set. The mean indicates the approximate position of the workpiece pose, while the covariance reflects the pose's uncertainty.

[0027] The particle filter framework outputs pose prediction parameters, including particle mean coordinates, which give the average position of the workpiece pose; covariance matrix, which is used to describe the distribution of pose; key feature point matching confidence, which reflects the reliability of key feature point matching; and occlusion area exclusion probability, which reflects the possibility of processing the occlusion area.

[0028] When constructing the particle filter framework, a uniformly distributed particle set is initialized. Each particle contains the workpiece's position coordinates and rotation angle in three-dimensional space. This ensures that hypotheses about the workpiece's pose are evenly distributed across the three-dimensional space. The end-effector pose corresponding to each particle is calculated using the forward solution of the robot's kinematics. This forward solution can be used to determine the end-effector's position and pose based on the particle's pose. The particle pose is projected onto the image plane based on the camera's extrinsic parameter matrix, generating predicted feature point coordinates. The camera's extrinsic parameter matrix enables the transformation from three-dimensional space to a two-dimensional image plane. The pixel error between the predicted feature point coordinates and the actual image feature point coordinates is calculated, and an observation likelihood function is constructed based on the sum of squared errors. The sum of squared errors measures the degree of difference between the predicted and actual feature points. A normalization process is performed to determine the weights of each particle. This process makes the weights comparable, retaining particles with weights greater than a preset threshold and removing particles with weights less than the threshold. This method selects more reliable particles and improves the accuracy of pose estimation.

[0029] Example 2: When constructing a multi-objective path planning model based on the workpiece pose prediction parameters, a multi-objective function for path planning is constructed, which includes two main optimization objective functions. The path length optimization objective function is obtained by dividing the path into segments, calculating the ratio of the straight-line distance of each segment to the corresponding movement speed, and then summing these ratios. This calculation method can comprehensively consider the impact of path length and movement speed on path planning. The joint motion optimization objective function is calculated by taking a weighted sum of the rotation angle, translational motion, and acceleration change of each joint. The weighted sum can be reasonably combined according to the importance of different motion parameters.

[0030] Based on the above multi-objective function, path planning constraints are constructed. These constraints are used to ensure the safety and feasibility of the robot arm during movement. The speed constraint is used to limit the movement speed range of the robot arm's end effector, so that the robot arm's movement speed is within a reasonable range, avoiding problems caused by too fast or too slow speed. The angle constraint is used to limit the range of variation of the rotation angle of each joint, ensuring that the movement of the joint is within a safe angle range and preventing damage caused by excessive rotation of the joint. The acceleration constraint is used to limit the value range of the robot arm's movement acceleration. Reasonable acceleration control can make the robot arm's movement smoother. The obstacle constraint is used to ensure a safe distance between the robot arm and surrounding fixed obstacles to prevent the robot arm from colliding with obstacles during movement.

[0031] The grasping path is encoded using a node sequence, with each node containing spatial coordinates and the robot's joint angles. This encoding method accurately describes every position and posture along the grasping path. A state transition probability is constructed based on the state transition cost and a heuristic evaluation value, which is then used to determine the next feasible node. The heuristic evaluation value is calculated by weighting the straight-line distance between nodes and the joint motion amplitude. This weighted calculation comprehensively considers the influence of distance and motion amplitude on node selection.

[0032] An adaptive state transition cost mechanism is introduced, which uses a time-varying weight coefficient that decays linearly with the number of planning iterations. This time-varying weight coefficient dynamically adjusts the state transition cost. In the early stages of planning, a larger weight coefficient can make the state transition cost have a greater impact on path selection. As the number of iterations increases, the weight coefficient gradually decreases, thereby achieving dynamic adjustment of the state transition cost. A dynamic heuristic evaluation mechanism is also introduced, which uses a time-varying influence factor that increases linearly with the number of planning iterations. This time-varying influence factor is used to adjust the weight of the heuristic evaluation value in the state transition probability. During the planning process, the weight of the heuristic evaluation value gradually increases with the number of iterations, making path planning more focused on heuristic information and improving path search efficiency.

[0033] Iterative optimization is performed based on an adaptive state transition cost mechanism and a dynamic heuristic evaluation mechanism. During each iteration, the generated path is evaluated for length cost and joint motion cost, and non-dominated solutions are added to the Pareto front solution set. The Pareto front solution set contains solutions that cannot simultaneously improve on both path length and joint motion amplitude objectives. These solutions are considered valid candidate solutions.

[0034] The optimal path is selected from the Pareto front solution set, based on the optimal solution that satisfies the length-motion trade-off. This optimal solution strikes a good balance between path length and joint motion amplitude, keeping the path as short as possible while minimizing joint motion amplitude. This optimal path is then smoothed using a spline curve. This smoothing process makes the path smoother, reduces impact and vibration during the robot's motion, and generates an optimal node sequence and corresponding robot motion parameters, which are used to guide the robot's actual motion.

[0035] When constructing the state transition cost, multiple calculations are required. First, the Euclidean distance between the current node and the next node is calculated as the path length cost. The Euclidean distance can intuitively reflect the distance between nodes. Then, the change in the rotation angle of each joint required for the robot arm to move from the current node to the next node is calculated as the joint motion cost. The size of the change in the rotation angle reflects the amplitude of the joint motion. Next, the minimum distance to the fixed obstacle in the path is calculated. If this minimum distance is less than the safe distance, the penalty cost is increased. The setting of the penalty cost can encourage path planning to avoid obstacles. Finally, the total state transition cost is generated by linearly combining the path length cost, joint motion cost, and penalty cost. The linear combination can integrate various cost factors to form a comprehensive state transition cost evaluation.

[0036] Example 3: When the strategy layer performs global grasp planning based on workpiece pose prediction parameters, it uses a parametric curve to describe the grasp path. This grasp path is expressed as a function of path parameters, where the path parameters range from 0 to 1 and the grasp path consists of X-axis, Y-axis, and Z-axis coordinate components. This parametric description allows the grasp path to be accurately expressed in three-dimensional space by varying the path parameters.

[0037] The grasping path is described using a quintic Bézier curve. Specifically, the spatial coordinates of the grasping path are calculated by multiplying the coordinates of the control vertices by the Bézier basis functions. The Bézier basis functions are calculated by multiplying the number of combinations by the power function of the path parameters. Using the quintic Bézier curve, a desired grasping path shape can be flexibly fitted. Adjusting the positions of the control vertices can change the path's direction and shape.

[0038] Construct the grasping path constraints, which are used to ensure the feasibility and safety of the grasping path. Position constraints are used to limit the coordinate range of the grasping path so that the path is within the spatial area that the robot can reach and meets the operation requirements. Speed ​​constraints are used to limit the range of the first-order derivative of the grasping path. The first-order derivative reflects the speed of change of the path. Reasonable limitation of the speed range can make the movement speed of the robot meet the requirements. Acceleration constraints are used to limit the range of the second-order derivative of the grasping path. The second-order derivative is related to the acceleration. Limiting the acceleration range can ensure the smoothness of the robot's movement and reduce impact. Curvature constraints are used to limit the range of curvature changes in the grasping path. Curvature reflects the degree of curvature of the path. Reasonable control of curvature changes can avoid excessive curvature of the path, so that the robot can move smoothly along the path.

[0039] A global multi-objective optimization function is constructed, consisting of a total path length term, a curvature integral term, a curvature change rate integral term, and an obstacle distance metric term. The total path length term measures the length of the path, the curvature integral term reflects the total degree of curvature of the path, the curvature change rate integral term reflects the speed of change in the path curvature, and the obstacle distance metric assesses the distance between the path and obstacles. Each term in the global multi-objective optimization function is weighted and combined using weighting coefficients. Different weights are assigned to each objective based on actual needs and their importance, allowing the optimization function to comprehensively consider multiple factors and achieve multi-objective optimization of the grasping path.

[0040] The grasping path interval is discretized into multiple path segments, and the global multi-objective optimization function is discretized to construct a global discretized objective function. The global discretized objective function includes path segment length, path segment curvature, curvature change, and minimum obstacle distance. Through discretization, the continuous path optimization problem is converted into a discrete problem, making it easier to solve using optimization algorithms.

[0041] A gradient descent method is used to iteratively optimize the global discretized objective function. The specific process is to calculate the gradient of the objective function with respect to the control vertex and then update the position coordinates of the control vertex along the negative gradient direction. By continuously iteratively updating the position of the control vertex, the value of the objective function is gradually reduced, thus finding a more optimal grasping path.

[0042] The optimized grasping path is smoothed using cubic spline interpolation. By maintaining the continuity of the position derivatives, velocity derivatives, and acceleration derivatives at the interpolation endpoints, a smooth and continuous grasping path is generated. Smoothing can make the path smoother and prevent vibration or impact caused by the uneven path during the movement of the robot arm. Based on the smooth and continuous grasping path, a node sequence and the corresponding robot arm movement speed are generated. The node sequence specifies the various positions that the robot arm needs to pass through during the grasping process, and the movement speed parameter provides a basis for the robot arm's motion control.

[0043] Throughout the global grasp planning process, from the parametric description of the grasping path to the application of Bezier curves, to the construction of various constraints and the setting of optimization functions, each step is interconnected and works together to generate the optimal grasping path. The use of parametric curves and quintic Bezier curves provides a flexible method for describing the grasping path. The setting of constraints ensures the feasibility and safety of the path. The construction of the multi-objective optimization function comprehensively considers several important factors such as path length, curvature, and obstacle distance. The iterative optimization process of the gradient descent method continuously improves the path, and cubic spline interpolation smoothing makes the path more suitable for the movement of the robot arm.

[0044] By strictly following the above steps, global grasp planning can be completed based on the workpiece pose prediction parameters, generating a desired grasping path and corresponding robotic arm motion parameters. During implementation, each link requires precise processing to ensure the accuracy of each step, enabling the strategy layer to effectively perform global grasp planning and provide reliable path guidance for subsequent robotic arm grasping operations. The entire process emphasizes detail and the logical relationships between each step to achieve efficient and accurate global grasp planning.

[0045] Example 4: When the adjustment layer performs local path corrections based on the optimal grasping path data, it constructs a local planning window based on the robot's current position, current joint angles, and current movement speed. The current position here refers to the actual coordinates of the robot in space, the current joint angles reflect the posture of each joint of the robot, and the current movement speed is the motion rate of the robot's end effector. The size of the local planning window is adaptively adjusted using the speed correlation coefficient, establishing a positive correlation between the size of the local planning window and the current movement speed. For example, when the robot moves at a higher speed, the size of the local planning window will increase accordingly to plan longer paths in advance; when the movement speed is lower, the size of the local planning window will decrease, focusing more on path corrections in the current vicinity.

[0046] Multi-view visual data is used to construct a local environment model. Multi-view visual data comes from multiple perspectives such as forward-looking cameras, side-looking cameras, and overhead cameras. The image data is transformed to obtain position data in the local coordinate system. Coordinate transformation is to convert image data taken by different cameras into a unified local coordinate system to make the data consistent and comparable. The occupancy probability of the grid map is updated based on the position data. The grid map divides the local environment into multiple small grid cells. Each grid cell has an occupancy probability value, which indicates the possibility that the position is occupied by an object. The occupancy probability value of each grid is calculated by the probability accumulation method. As new visual data is continuously input, the occupancy probability of the grid will be updated according to the probability accumulation rule, thereby gradually constructing an accurate local environment model.

[0047] The Kalman filter algorithm is used to track dynamic obstacles. The Kalman filter algorithm is a commonly used state estimation method that uses a state prediction equation to predict the obstacle's state vector, which includes information such as the obstacle's position and velocity. The predicted state is updated based on measurement data. The measurement data comes from multi-view camera observations of the obstacle. By combining the predicted state with the measurement data, the obstacle's precise position and motion information is obtained. For example, when a dynamic obstacle moves in the scene, the Kalman filter algorithm predicts its next position based on its previous motion state. It then uses the camera's observation data to correct the prediction, accurately tracking the obstacle's trajectory.

[0048] Construct a path correction model. Use the kinematic equations of the robotic arm as state equations. The state equations include spatial coordinates, movement speed, and joint angles. These state variables describe the motion state of the robotic arm. Construct state constraints and dynamic constraints. State constraints are used to limit the range of spatial coordinate values ​​and the range of movement speed values ​​to ensure that the motion state of the robotic arm is within a safe and feasible range. Dynamic constraints are used to limit the range of angular velocity, angular acceleration, and joint angle change rate to ensure that the motion of the robotic arm conforms to dynamic characteristics and avoid excessive acceleration or velocity changes.

[0049] A multi-objective cost function is then constructed, which includes a reference trajectory tracking term, an obstacle avoidance term, a trajectory smoothing term, and an energy consumption term. The reference trajectory tracking term is used to measure the degree of deviation between the corrected path and the optimal grasping path, hoping to keep it as close to the reference trajectory as possible. The obstacle avoidance term is used to evaluate the distance between the path and obstacles to ensure that the path avoids obstacles. The trajectory smoothing term is used to ensure the smoothness of the path and reduce vibration during the movement of the robot arm. The energy consumption term is used to consider the energy consumption of the robot arm movement and pursue energy optimization. The various items in the multi-objective cost function are weighted and combined using weighting coefficients. Different weights are determined based on actual needs and the importance of each objective, so that the cost function can comprehensively reflect multiple optimization objectives.

[0050] The Lagrange multiplier method is used to optimize and solve the multi-objective cost function. This method is an effective approach for solving constrained optimization problems. By constructing a Lagrange function and introducing constraints, the constrained optimization problem is transformed into an unconstrained one. By solving the system of partial derivative equations, the optimal control variables, including the control parameters of each joint of the manipulator, are obtained and used to guide the correction of the manipulator's motion path.

[0051] Throughout the local path correction process, each step works closely together. Constructing a local planning window enables the system to dynamically adjust the planning range based on the robot's current motion state. The local environment model constructed from multi-view visual data provides accurate environmental information for path correction. The Kalman filter algorithm's tracking of dynamic obstacles ensures the system can promptly respond to dynamic changes in the environment. The path correction model and multi-objective cost function provide the mathematical model and optimization objectives for path correction. The Lagrange multiplier method's optimization solution ultimately yields a feasible path correction solution.

[0052] For example, when the robotic arm is moving along the optimal grasping path and suddenly detects a dynamic obstacle in front of it (such as another moving robotic arm), the local planning window will adaptively adjust its size according to the current moving speed. The multi-view camera obtains visual data of the obstacle and updates the occupancy probability of the grid map through coordinate transformation. The Kalman filter algorithm tracks the movement trajectory of the obstacle. The path correction model constructs state equations and constraints based on the current state of the robotic arm and obstacle information. The multi-objective cost function comprehensively considers factors such as reference trajectory tracking, obstacle avoidance, trajectory smoothing, and energy consumption. Finally, the optimal control quantity is solved through the Lagrange multiplier method, so that the robotic arm can avoid obstacles in time while staying as close to the original optimal grasping path as possible.

[0053] This implementation allows the adjustment layer to effectively perform local path corrections based on the optimal grasping path data, enabling the robot arm to safely and reliably complete grasping tasks in complex working environments and cope with sudden environmental changes and obstacles. The entire process focuses on practical application scenarios and the robot's motion characteristics, achieving precise local path corrections through a series of processing steps and algorithms.

[0054] Example 5: When the execution layer implements trajectory tracking for the robotic arm based on a visual servo control algorithm, a six-degree-of-freedom kinematic model of the robotic arm must be established. This model includes translational and rotational equations. The translational equation includes driving force, friction, and gravity terms. The driving force term is the source of the force that drives the robotic arm's translational motion, the friction term reflects the frictional resistance encountered during motion, and the gravity term is the influence of the robotic arm's own weight. The rotational equation includes torque and moment of inertia terms. The torque term is the torque that causes the robotic arm's joints to rotate, while the moment of inertia term is related to the robotic arm's moment of inertia and angular acceleration, describing the inertial characteristics of the robotic arm's rotational motion.

[0055] The six-degree-of-freedom kinematic model is constructed as a state-space expression. The state vectors of this state-space expression include spatial coordinates, movement velocity, joint angles, and angular velocity. These state vectors fully describe the motion state of the robot arm. Spatial coordinates represent the position of the robot end in three-dimensional space, movement velocity is the rate of motion of the end effector, joint angles reflect the posture of each joint, and angular velocity is the speed of joint rotation. The control vectors of this state-space expression include driving force and three-axis torque. The driving force controls the translational motion of the robot arm, while the three-axis torque controls the rotational motion of the joints.

[0056] Linearize the state-space expression, calculate the partial derivatives of the system state equation with respect to the state vector and the control vector, and construct a linearized prediction model through the calculation of partial derivatives. Linearization is used to approximate complex nonlinear systems into linear systems, facilitating subsequent analysis and control.

[0057] A trajectory tracking prediction cost function is constructed, consisting of a tracking error term, a control penalty term, and a control increment penalty term. The tracking error term measures the deviation between the robot's actual trajectory and the desired trajectory. The control penalty term penalizes excessive control inputs, and the control increment penalty term limits drastic changes in the control variable. Each penalty term is weighted and combined using a weight matrix. Appropriate weights are assigned to different penalty terms based on actual control requirements to balance the importance of each objective.

[0058] Various constraints are then constructed. State constraints include spatial coordinate constraints and joint angle constraints. Spatial coordinate constraints limit the spatial range that the end-of-arm can reach, while joint angle constraints ensure that the rotation angles of each joint are within a safe range. Control constraints include driving force constraints and torque constraints. The driving force constraint specifies the range of driving force values, while the torque constraint limits the magnitude of torque on each axis. Control increment constraints include driving force increment constraints and torque increment constraints, which are used to limit the variation of driving force and torque to avoid sudden changes in the control quantity.

[0059] The trajectory tracking prediction cost function is converted into the standard form of a quadratic programming problem. The quadratic form matrix and linear term coefficients are calculated, and the inequality constraint matrix and equality constraint matrix are constructed. The standard form of the quadratic programming problem facilitates the use of specific optimization algorithms to solve it. Matrix operations are used to transform the problem into a mathematically tractable form.

[0060] The active set method is used to solve quadratic programming problems. By identifying valid constraints and solving subproblems, the optimal solution is gradually approached. The active set method is a common algorithm for solving quadratic programming problems. It can find the optimal control variable while considering the constraints.

[0061] Based on the optimization results, control variables are mapped and the total driving force is distributed to each drive motor using the driving force distribution matrix. This matrix rationally distributes the total driving force to each drive motor based on the structure and kinematics of the robotic arm. The three-axis torque is distributed to each joint driver using the torque distribution matrix. This matrix ensures that the three-axis torque can be correctly applied to each joint, achieving control of joint rotation.

[0062] The output of the drive motor is limited, limiting the output torque according to the motor's rated power to prevent damage caused by overload. The rotation angle is limited according to the joint's maximum rotation speed to ensure that the joint's rotation speed is within a safe range, preventing vibration or damage to the robot arm caused by excessive speed. After limiting, the final grasping control command is generated to control the actual movement of the robot arm.

[0063] For example, when a six-axis robotic arm grasps a workpiece, the execution layer first establishes a six-degree-of-freedom kinematic model, taking into account the driving force, friction, and gravity in the translation equation, as well as the torque and moment of inertia in the rotation equation. After constructing the model as a state-space expression, it is linearized to obtain a linearized prediction model. A trajectory tracking prediction cost function is constructed, where the tracking error term focuses on whether the robotic arm accurately follows the preset grasping path, the control quantity penalty term avoids excessive driving force and torque input, and the control increment penalty term prevents drastic changes in the control quantity.

[0064] Assume that during the movement, the actual trajectory of the robot arm deviates from the desired trajectory. For example, the end effector deviates from the preset path by 5 mm in the X-axis direction. At this time, the value of the tracking error term will increase. At the same time, if the current control variable causes the rotation angle of a joint to approach its maximum limit, the control constraint will be triggered. The quadratic programming problem is solved by the effective set method to obtain the adjusted control variable, such as increasing the driving force in the X-axis direction and adjusting the torque of the relevant joints to return the robot arm to the desired trajectory. The driving force distribution matrix distributes the total driving force to the corresponding drive motors, and the torque distribution matrix distributes the three-axis torque to each joint driver. Finally, the motor output is limited to ensure that the motor and joint operate within a safe range. The final grasping control command is generated to enable the robot arm to accurately reach the target position and grasp the workpiece.

[0065] From the establishment of the kinematic model to the generation of control instructions, each step is closely centered around the trajectory tracking control of the robotic arm. Through precise modeling of the robotic arm's motion state, linearization processing, cost function construction, constraint setting, and optimization solution, precise control of the robotic arm's motion is achieved. In practical applications, whether facing simple linear motion or complex curved trajectories, this implementation method can ensure that the robotic arm moves accurately along the preset trajectory and completes the workpiece grasping task through a series of calculation and control steps. At the same time, through various constraints and limiting processing, the safety and reliability of the robotic arm's motion are guaranteed, avoiding equipment damage or task failure due to improper control.

[0066] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0067] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A workpiece grasping method based on visual feedback, characterized in that: include: Acquire real-time image data of the target workpiece through a multi-view visual perception device, wherein the multi-view visual perception device includes a front-view camera, a side-view camera, and a top-view camera; Performing feature extraction processing on the real-time image data based on a feature fusion network to obtain workpiece surface feature data; inputting the workpiece surface feature data into a pre-trained pose estimation model, wherein the pose estimation model adopts a particle filter framework, iteratively updates the workpiece pose based on an observation likelihood function, and generates workpiece pose prediction parameters; A multi-objective path planning model is constructed based on the workpiece posture prediction parameters, wherein the multi-objective path planning model takes the shortest grasping path length and the smallest joint motion amplitude as optimization goals, and globally optimizes the grasping path using a dynamic programming algorithm, wherein the dynamic programming algorithm introduces a state transition cost and a heuristic evaluation function; and optimal grasping path data is output based on the multi-objective path planning model; A hierarchical grasping control model is established based on the optimal grasping path data, and the hierarchical grasping control model includes a strategy layer, an adjustment layer and an execution layer, wherein the strategy layer performs global grasping planning based on the workpiece posture prediction parameters, the adjustment layer performs local path correction based on the optimal grasping path data, and the execution layer realizes the trajectory tracking of the robot arm based on the visual servo control algorithm; the hierarchical grasping control model outputs grasping control instructions to realize visual feedback grasping control of the target workpiece.

2. The workpiece grasping method based on visual feedback according to claim 1, characterized in that: The workpiece surface feature data is input into a pre-trained pose estimation model. The pose estimation model uses a particle filter framework to iteratively update the workpiece pose based on the observation likelihood function to generate workpiece pose prediction parameters including: Acquire real-time image data, the real-time image data including workpiece edge contour information, surface texture information, key feature point coordinates, and background environmental occlusion information; construct a state space based on the real-time image data, and construct an action space based on the translational motion, rotational angle, and gripper opening and closing degree executable by the end effector of the robotic arm; Constructing a multidimensional observation likelihood function based on the state space and the action space, the multidimensional observation likelihood function including a contour matching term, a texture alignment term, a feature point coincidence term, and an occlusion exclusion term, wherein the contour matching term is calculated by the shape similarity between the actual contour of the workpiece and the predicted contour, the texture alignment term is calculated by the grayscale correlation between the workpiece surface texture and the template texture, the feature point coincidence term is calculated by the position deviation between the actual feature point coordinates and the predicted feature point coordinates, and the occlusion exclusion term is calculated by the pixel separation between the background area and the workpiece area; Constructing a particle filter framework, the particle filter framework includes an initial particle set, a state transfer function, and an observation likelihood calculation module. The initial particle set contains multiple workpiece pose hypothesis samples. The state transfer function predicts and updates the particle poses using a robotic arm kinematic model. The observation likelihood calculation module evaluates particle weights based on the multidimensional observation likelihood function. The initial particle set is updated by a resampling method. High-weight particles are retained through roulette wheel selection, low-weight particles are eliminated, and new particles are added. The mean and covariance of the workpiece pose are calculated based on the updated particle set. The pose prediction parameters are output based on the particle filter framework. The pose prediction parameters include particle mean coordinates, covariance matrix, key feature point matching confidence, and occlusion area exclusion probability.

3. The workpiece grasping method based on visual feedback according to claim 1, characterized in that: A multi-objective path planning model is constructed based on the workpiece posture prediction parameters. The multi-objective path planning model takes the shortest grasping path length and the smallest joint motion amplitude as optimization goals, and uses a dynamic programming algorithm to perform global optimization on the grasping path. The optimal grasping path data output based on the multi-objective path planning model includes: Constructing a multi-objective function for path planning, the multi-objective function including a path length optimization objective function and a joint motion optimization objective function, wherein the path length optimization objective function is calculated by summing the ratios of the straight-line distances of the path segments to the corresponding movement speeds, and the joint motion optimization objective function is calculated by weighted summing of the rotation angles, translational motions, and acceleration changes of each joint; Constructing path planning constraints based on the multi-objective function, the path planning constraints include speed constraints, angle constraints, acceleration constraints, and obstacle constraints. The speed constraints are used to limit the range of movement speed of the end effector of the manipulator, the angle constraints are used to limit the range of variation of the rotation angles of each joint, the acceleration constraints are used to limit the range of values ​​of the manipulator's motion acceleration, and the obstacle constraints are used to ensure a safe distance between the manipulator and surrounding fixed obstacles. The grasping path is encoded using a node sequence, where each node contains spatial coordinates and robot arm joint angle information. A state transition probability is constructed based on the state transition cost and a heuristic evaluation value, and the next feasible node is determined using the state transition probability. The heuristic evaluation value is calculated by weighting the straight-line distance between nodes and the joint motion amplitude. Introducing an adaptive state transfer cost mechanism, the adaptive state transfer cost mechanism adopts a time-varying weight coefficient, the time-varying weight coefficient changes linearly with the increase of the number of planning iterations, and the state transfer cost is dynamically adjusted by the time-varying weight coefficient; A dynamic heuristic evaluation mechanism is introduced. The dynamic heuristic evaluation mechanism uses a time-varying influence factor that increases linearly with the number of planning iterations. The time-varying influence factor is used to adjust the weight of the heuristic evaluation value in the state transition probability. Iterative optimization is performed based on the adaptive state transition cost mechanism and the dynamic heuristic evaluation mechanism. The length cost and joint motion cost of the path generated by each iteration are evaluated, and non-dominated solutions are added to the Pareto front solution set. An optimal solution that satisfies the length-motion trade-off is selected from the Pareto front solution set as the optimal path, and a spline curve smoothing process is performed on the optimal path to generate an optimal node sequence and corresponding robot arm motion parameters.

4. The workpiece grasping method based on visual feedback according to claim 1, characterized in that: The strategy layer performs global grasping planning based on the workpiece pose prediction parameters, including: A parameterized curve is used to describe the grasping path, and the grasping path is expressed as a function of a path parameter, wherein the value of the path parameter ranges from 0 to 1, and the grasping path includes an X-axis coordinate component, a Y-axis coordinate component, and a Z-axis coordinate component; The grasping path is described based on a quintic Bezier curve, and the spatial coordinates of the grasping path are calculated by summing the products of the control vertex coordinates and the Bezier basis functions, wherein the Bezier basis functions are calculated by the power function of the combination number and the path parameter; Constructing grasping path constraints, the grasping path constraints including position constraints, velocity constraints, acceleration constraints, and curvature constraints. The position constraint is used to limit the coordinate range of the grasping path, the velocity constraint is used to limit the first-order derivative range of the grasping path, the acceleration constraint is used to limit the second-order derivative range of the grasping path, and the curvature constraint is used to limit the curvature variation range of the grasping path. Constructing a global multi-objective optimization function, the global multi-objective optimization function including a total path length term, a curvature integral term, a curvature change rate integral term, and an obstacle distance measurement term, and performing a weighted combination of each term in the global multi-objective optimization function using a weighting coefficient; Discretize the grasping path interval into multiple path segments, discretize the global multi-objective optimization function, and construct a global discretization objective function, wherein the global discretization objective function includes path segment length, path segment curvature, curvature change, and minimum obstacle distance; Iteratively optimizing the global discretized objective function using a gradient descent method, and updating the position coordinates of the control vertex along the negative gradient direction by calculating the gradient value of the objective function with respect to the control vertex; The optimized grasping path is smoothed by cubic spline interpolation. By maintaining the continuity of the position derivative, velocity derivative, and acceleration derivative at the interpolation endpoints, a smooth and continuous grasping path is generated. A node sequence and the corresponding robot arm movement speed are generated based on the smooth and continuous grasping path.

5. The workpiece grasping method based on visual feedback according to claim 1, characterized in that: The adjustment layer performs local path correction based on the optimal crawling path data, including: A local planning window is constructed based on the current position, current joint angle, and current movement speed of the robot arm, and the size of the local planning window is adaptively adjusted using a speed correlation coefficient to establish a positive correlation between the size of the local planning window and the current movement speed; A local environment model is constructed using multi-view visual data, and the image data is transformed to obtain position data in the local coordinate system. The occupancy probability of the grid map is updated based on the position data, and the occupancy probability value of each grid is calculated using the probability accumulation method. The Kalman filter algorithm is used to track dynamic obstacles. The state vector of the obstacle is predicted through the state prediction equation. The predicted state is updated based on the measurement data to obtain the precise position and motion information of the obstacle. Constructing a path correction model, using the kinematic equation of the manipulator as a state equation, the state equation including spatial coordinates, movement speed, and joint angles, and constructing state constraints and dynamic constraints. The state constraints are used to limit the value ranges of the spatial coordinates and the movement speed, and the dynamic constraints are used to limit the value ranges of the angular velocity, angular acceleration, and the rate of change of the joint angles; Constructing a multi-objective cost function, the multi-objective cost function including a reference trajectory tracking term, an obstacle avoidance term, a trajectory smoothing term, and an energy consumption term, and performing a weighted combination of each term in the multi-objective cost function using a weighting coefficient; The Lagrange multiplier method is used to optimize and solve the multi-objective cost function, a Lagrange function is constructed and constraints are introduced, and the optimal control quantity is obtained by solving the partial derivative equation group.

6. The workpiece grasping method based on visual feedback according to claim 1, characterized in that: The execution layer implements trajectory tracking of the robot arm based on the visual servo control algorithm and outputs grasping control instructions through the hierarchical grasping control model to achieve visual feedback grasping control of the target workpiece, including: Establishing a six-degree-of-freedom kinematic model for the robotic arm, the six-degree-of-freedom kinematic model includes a translation equation and a rotation equation, the translation equation includes a driving force term, a friction term, and a gravity term, and the rotation equation includes a torque term and a moment of inertia term; Constructing the six-degree-of-freedom kinematic model into a state-space expression, wherein the state vector of the state-space expression includes spatial coordinates, movement speed, joint angles, and angular velocity, and the control vector of the state-space expression includes driving force and three-axis torque; Linearizing the state space expression, calculating the partial derivatives of the system state equation with respect to the state vector and the control vector, and constructing a linearized prediction model; Constructing a trajectory tracking prediction cost function, wherein the trajectory tracking prediction cost function includes a tracking error term, a control amount penalty term, and a control increment penalty term, and performing a weighted combination of each penalty term through a weight matrix; Constructing state constraints, including space coordinate constraints and joint angle constraints, constructing control constraints, including driving force constraints and torque constraints, and constructing control increment constraints, including driving force increment constraints and torque increment constraints; Converting the trajectory tracking prediction cost function into a standard form of a quadratic programming problem, calculating the quadratic matrix and the linear term coefficients, and constructing the inequality constraint matrix and the equality constraint matrix; The effective set method is used to solve the quadratic programming problem, and the optimal solution is gradually approached by identifying effective constraints and solving sub-problems; Based on the optimization solution, control quantity mapping is performed, the total driving force is distributed to each driving motor through the driving force distribution matrix, and the three-axis torque is distributed to each joint driver through the torque distribution matrix; The output of the drive motor is limited, the output torque is limited according to the rated power of the motor, and the rotation angle is limited according to the maximum rotation speed of the joint to generate the final grasping control instruction.

7. The workpiece grasping method based on visual feedback according to claim 1, characterized in that: The real-time image data is subjected to feature extraction processing based on a feature fusion network to obtain workpiece surface feature data, including: grayscale processing of the real-time image data acquired by a front-view camera, a side-view camera, and a top-view camera respectively to obtain single-channel grayscale images; Gaussian filtering and denoising of each grayscale image, calculating the local mean and variance through a sliding window, and adjusting the image contrast; using a scale-invariant feature transformation algorithm to extract local feature points of each image, establishing a correspondence between images of different viewpoints through feature point matching; converting the matched feature point coordinates to a unified world coordinate system, fusing multi-view feature point information through a weighted average method, and generating workpiece surface feature data containing depth information.

8. The workpiece grasping method based on visual feedback according to claim 2, characterized in that: Constructing a particle filter framework includes: initializing a particle set with uniform distribution, where each particle contains the position coordinates and rotation angle of the workpiece in three-dimensional space; calculating the end effector pose corresponding to the particle through the forward solution of the robot arm kinematics, projecting the particle pose onto the image plane based on the camera extrinsic parameter matrix, and generating predicted feature point coordinates; calculating the pixel error between the predicted feature point coordinates and the actual image feature point coordinates, and constructing the observation likelihood function based on the sum of squared errors; obtaining the weight value of each particle through normalization processing, retaining particles with weight values ​​greater than a preset threshold, and eliminating particles with weight values ​​less than the preset threshold.

9. The workpiece grasping method based on visual feedback according to claim 3, characterized in that: Constructing the state transition cost includes: calculating the Euclidean distance between the current node and the next node as the path length cost; calculating the change in the rotation angle of each joint required for the robot arm to move from the current node to the next node as the joint motion cost; calculating the minimum distance to the fixed obstacle in the path, and increasing the penalty cost if it is less than the safety distance; generating the total state transition cost by linearly combining the path length cost, joint motion cost and penalty cost.

10. A workpiece grasping system based on visual feedback, used to implement the workpiece grasping method based on visual feedback according to any one of claims 1 to 9, characterized in that: include: A first module is configured to acquire real-time image data of a target workpiece through a multi-view visual perception device, wherein the multi-view visual perception device includes a front-view camera, a side-view camera, and a top-view camera; Performing feature extraction processing on the real-time image data based on a feature fusion network to obtain workpiece surface feature data; inputting the workpiece surface feature data into a pre-trained pose estimation model, wherein the pose estimation model adopts a particle filter framework, iteratively updates the workpiece pose based on an observation likelihood function, and generates workpiece pose prediction parameters; The second module is used to construct a multi-objective path planning model based on the workpiece posture prediction parameters. The multi-objective path planning model takes the shortest grasping path length and the smallest joint motion amplitude as optimization goals, and uses a dynamic programming algorithm to globally optimize the grasping path. The dynamic programming algorithm introduces state transition cost and heuristic evaluation function to improve path search efficiency; and outputs optimal grasping path data based on the multi-objective path planning model; The third module is used to establish a hierarchical grasping control model based on the optimal grasping path data. The hierarchical grasping control model includes a strategy layer, an adjustment layer and an execution layer. The strategy layer performs global grasping planning based on the workpiece posture prediction parameters, the adjustment layer performs local path correction based on the optimal grasping path data, and the execution layer realizes the trajectory tracking of the robot arm based on the visual servo control algorithm; the hierarchical grasping control model outputs grasping control instructions to realize visual feedback grasping control of the target workpiece.

Images