Workpiece grabbing method and system based on visual feedback

Abstract

The application relates to the technical field of automatic workpiece grabbing and visual servo control, and discloses a workpiece grabbing method and system based on visual feedback, which comprises the following steps: acquiring real-time images through front-view, side-view and overhead multi-view cameras, and extracting surface features through a feature fusion network; inputting the features into a pose estimation model based on a particle filtering framework, iteratively updating the pose in combination with a multi-dimensional observation likelihood function, and generating predicted parameters; constructing a multi-target path planning model based on the parameters, taking the shortest path and the minimum joint motion as targets, and optimizing the path by using a dynamic programming algorithm; establishing a hierarchical control model, performing global planning in the strategy layer, correcting the local path in the adjustment layer, realizing trajectory tracking by visual servo control in the execution layer, and outputting control instructions to complete grabbing; and through multi-view perception, robust pose estimation, global optimization path planning and hierarchical control, the precision, efficiency and stability of workpiece grabbing in a complex environment are improved.

Description

Technical Field

[0001] This invention relates to the field of automated workpiece gripping and vision servo control technology, specifically a workpiece gripping method and system based on visual feedback. Background Technology

[0002] In industrial automation production, logistics sorting, and intelligent manufacturing scenarios, workpiece gripping is a crucial step in achieving automated processes, and its efficiency and accuracy directly impact overall production effectiveness. Traditional workpiece gripping methods often rely on preset programs or fixed trajectory control, making it difficult to adapt to dynamic scenarios with varying workpiece postures and complex environments. For example, on assembly lines, workpieces may shift positions due to conveyor vibrations, or stacking or obstruction may make gripping point identification difficult. Traditional robotic arms often suffer from gripping failures and workpiece damage due to a lack of real-time sensing and adjustment capabilities.

[0003] With the development of machine vision technology, grasping methods based on visual feedback have gradually become a research hotspot. However, existing technologies still have many limitations: First, single-view vision devices acquire limited image information, making it difficult to fully reconstruct the three-dimensional shape of the workpiece, especially when there are complex textures or occlusions on the workpiece surface, which can easily lead to incomplete feature extraction and affect the accuracy of pose estimation; Second, pose estimation models often use single feature matching methods, such as relying only on edge contours or key feature points, ignoring multi-dimensional information such as texture and occlusion, resulting in insufficient robustness of pose prediction in dynamic environments; Third, path planning often focuses on a single objective (such as the shortest path) as the optimization direction, without considering factors such as the range of motion of the robotic arm joints and energy consumption, which can easily lead to joint movement exceeding limits or path jitter, reducing grasping stability; Fourth, control models are mostly single-level structures, making it difficult to balance global planning and local correction, and unable to quickly adjust the trajectory when encountering sudden obstacles or path deviations, resulting in decreased tracking accuracy.

[0004] In existing technologies, multi-view feature fusion often suffers from insufficient viewpoint bias correction, leading to discrepancies between the fused features and the actual workpiece shape. Particle filtering frameworks fail to effectively introduce multi-dimensional observation constraints during particle degradation, resulting in slow pose estimation convergence. Dynamic programming algorithms lack adaptive cost adjustment mechanisms during state transitions, leading to inefficient path search. High coupling between the policy layer and execution layer in hierarchical control hinders rapid response. These issues collectively restrict the application of visual feedback grasping technology in high-precision, high-dynamic scenarios, necessitating a comprehensive solution integrating multi-view perception, robust pose estimation, global optimization path planning, and hierarchical control. Summary of the Invention

[0005] The purpose of this invention is to provide a workpiece gripping method and system based on visual feedback to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a workpiece gripping method based on visual feedback, the method comprising:

[0007] Real-time image data of the target workpiece is acquired through a multi-view visual perception device, which includes a front-view camera, a side-view camera, and a top-view camera. Feature extraction processing is performed on the real-time image data based on a feature fusion network to obtain workpiece surface feature data. The workpiece surface feature data is then input into a pre-trained pose estimation model, which uses a particle filter framework and iteratively updates the workpiece pose based on the observation likelihood function to generate workpiece pose prediction parameters.

[0008] A multi-objective path planning model is constructed based on the workpiece pose prediction parameters. The multi-objective path planning model aims to minimize the gripping path length and the joint motion amplitude. A dynamic programming algorithm is used to globally optimize the gripping path. The dynamic programming algorithm introduces state transition costs and heuristic evaluation functions. The optimal gripping path data is output based on the multi-objective path planning model.

[0009] A hierarchical grasping control model is established based on the optimal grasping path data. The hierarchical grasping control model includes a strategy layer, an adjustment layer, and an execution layer. The strategy layer performs global grasping planning based on the workpiece pose prediction parameters, the adjustment layer performs local path correction based on the optimal grasping path data, and the execution layer realizes the trajectory tracking of the robotic arm based on a visual servo control algorithm. The hierarchical grasping control model outputs grasping control commands to achieve visual feedback grasping control of the target workpiece.

[0010] Preferably, the workpiece surface feature data is input into a pre-trained pose estimation model. This model employs a particle filter framework and iteratively updates the workpiece pose based on the observation likelihood function to generate workpiece pose prediction parameters, including:

[0011] Acquire real-time image data, which includes workpiece edge contour information, surface texture information, key feature point coordinates, and background environment occlusion information; construct a state space based on the real-time image data, and construct an action space based on the translational motion, rotational angle, and gripper opening and closing degree that the robotic arm end effector can execute;

[0012] A multi-dimensional observation likelihood function is constructed based on the state space and the action space. The multi-dimensional observation likelihood function includes a contour matching term, a texture alignment term, a feature point coincidence term, and an occlusion exclusion term. The contour matching term is calculated by the shape similarity between the actual contour of the workpiece and the predicted contour. The texture alignment term is calculated by the grayscale correlation between the workpiece surface texture and the template texture. The feature point coincidence term is calculated by the positional deviation between the actual feature point coordinates and the predicted feature point coordinates. The occlusion exclusion term is calculated by the pixel separation degree between the background region and the workpiece region.

[0013] A particle filtering framework is constructed, which includes an initial particle set, a state transition function, and an observation likelihood calculation module. The initial particle set contains multiple workpiece pose hypothesis samples. The state transition function predicts and updates the particle pose through the robot arm kinematics model. The observation likelihood calculation module evaluates the particle weights based on the multi-dimensional observation likelihood function.

[0014] The initial particle set is updated using a resampling method. High-weight particles are retained and low-weight particles are removed and new particles are added by roulette wheel selection. The mean and covariance of the workpiece pose are calculated based on the updated particle set. Based on the particle filtering framework, pose prediction parameters are output, including particle mean coordinates, covariance matrix, key feature point matching confidence, and occlusion region exclusion probability.

[0015] Preferably, a multi-objective path planning model is constructed based on the workpiece pose prediction parameters. The multi-objective path planning model aims to minimize the grasping path length and joint motion amplitude. A dynamic programming algorithm is used to globally optimize the grasping path. The optimal grasping path data output based on the multi-objective path planning model includes:

[0016] A multi-objective function for path planning is constructed, which includes a path length optimization objective function and a joint motion optimization objective function. The path length optimization objective function is calculated by summing the ratios of the straight-line distances of the path segments to the corresponding movement speeds. The joint motion optimization objective function is calculated by weighted summing of the rotation angles, translational motions, and acceleration changes of each joint.

[0017] The path planning constraints are constructed based on the multi-objective function. The path planning constraints include speed constraints, angle constraints, acceleration constraints, and obstacle constraints. The speed constraints are used to limit the range of movement speed of the end effector of the robotic arm. The angle constraints are used to limit the range of rotation angle of each joint. The acceleration constraints are used to limit the range of values ​​of the robotic arm's motion acceleration. The obstacle constraints are used to ensure a safe distance between the robotic arm and surrounding fixed obstacles.

[0018] The grasping path is encoded using a node sequence. Each node contains spatial coordinates and robotic arm joint angle information. A state transition probability is constructed based on the state transition cost and a heuristic evaluation value. The next feasible node is determined by the state transition probability. The heuristic evaluation value is obtained by weighting the straight-line distance between nodes and the joint motion amplitude.

[0019] An adaptive state transition cost mechanism is introduced, which adopts a time-varying weight coefficient. The time-varying weight coefficient decreases linearly with the number of planning iterations, and the state transition cost is dynamically adjusted through the time-varying weight coefficient.

[0020] A dynamic heuristic evaluation mechanism is introduced, which adopts a time-varying influence factor. The time-varying influence factor increases linearly with the number of planning iterations. The weight of the heuristic evaluation value in the state transition probability is adjusted by the time-varying influence factor. Iterative optimization is performed based on the adaptive state transition cost mechanism and the dynamic heuristic evaluation mechanism. The length cost and joint motion cost of the path generated in each iteration are evaluated, and the non-dominated solution is added to the Pareto front solution set.

[0021] The optimal solution that satisfies the length-motion tradeoff is selected from the Pareto front solution set as the optimal path. The optimal path is then smoothed using spline curves to generate the optimal node sequence and the corresponding robotic arm motion parameters.

[0022] Preferably, the strategy layer performs global grasping planning based on the workpiece pose prediction parameters, including:

[0023] The crawling path is described by a parametric curve, and the crawling path is represented as a function of path parameters. The path parameters range from 0 to 1, and the crawling path includes X-axis coordinate components, Y-axis coordinate components, and Z-axis coordinate components.

[0024] The crawling path is described based on a quintic Bézier curve. The spatial coordinates of the crawling path are obtained by summing the product of the vertex coordinates and the Bézier basis functions. The Bézier basis functions are calculated by combining the number of combinations and the power function of the path parameters.

[0025] Construct gripping path constraints, which include position constraints, velocity constraints, acceleration constraints, and curvature constraints. The position constraints are used to limit the coordinate range of the gripping path, the velocity constraints are used to limit the range of the first derivative of the gripping path, the acceleration constraints are used to limit the range of the second derivative of the gripping path, and the curvature constraints are used to limit the range of curvature variation of the gripping path.

[0026] A global multi-objective optimization function is constructed, which includes a total path length term, a curvature integral term, a curvature change rate integral term, and an obstacle distance metric term. The terms in the global multi-objective optimization function are weighted and combined using weighting coefficients.

[0027] The capture path interval is discretized into multiple path segments, and the global multi-objective optimization function is discretized to construct a global discretized objective function. The global discretized objective function includes path segment length, path segment curvature, curvature change, and minimum distance to obstacles.

[0028] The global discretized objective function is iteratively optimized using the gradient descent method. The position coordinates of the control vertex are updated along the negative gradient direction by calculating the gradient value of the objective function with respect to the control vertex.

[0029] The optimized grasping path is smoothed by cubic spline interpolation. By maintaining the continuity of the position derivative, velocity derivative, and acceleration derivative at the interpolation endpoints, a smooth and continuous grasping path is generated. Based on the smooth and continuous grasping path, a node sequence and the corresponding robotic arm movement speed are generated.

[0030] Preferably, the adjustment layer performs local path correction based on the optimal crawling path data, including:

[0031] A local planning window is constructed based on the current position, current joint angle, and current moving speed of the robotic arm. The size of the local planning window is adaptively adjusted through a speed correlation coefficient, and a positive correlation is established between the size of the local planning window and the magnitude of the current moving speed.

[0032] A local environment model is constructed using multi-view visual data. The image data is transformed to obtain the location data in the local coordinate system. The occupancy probability of the raster map is updated based on the location data. The occupancy probability value of each raster is calculated by the probability accumulation method.

[0033] The Kalman filter algorithm is used to track dynamic obstacles. The state vector of the obstacle is predicted by the state prediction equation. The predicted state is updated based on the measurement data to obtain the precise position and motion information of the obstacle.

[0034] A path correction model is constructed, and the kinematic equation of the robotic arm is used as the state equation. The state equation includes spatial coordinates, movement speed and joint angle. State constraints and dynamic constraints are constructed. The state constraints are used to limit the range of values ​​for spatial coordinates and movement speed. The dynamic constraints are used to limit the range of values ​​for angular velocity, angular acceleration and joint angle change rate.

[0035] A multi-objective cost function is constructed, which includes a reference trajectory tracking term, an obstacle avoidance term, a trajectory smoothing term, and an energy consumption term. The terms in the multi-objective cost function are weighted and combined using weighting coefficients.

[0036] The Lagrange multiplier method is used to optimize the multi-objective cost function. A Lagrange function is constructed and constraints are introduced. The optimal control quantity is obtained by solving the partial derivative equations.

[0037] Preferably, the execution layer uses a visual servo control algorithm to track the trajectory of the robotic arm, and outputs grasping control commands through the hierarchical grasping control model to achieve visual feedback grasping control of the target workpiece, including:

[0038] A six-degree-of-freedom kinematic model of a robotic arm is established. The six-degree-of-freedom kinematic model includes translation equations and rotation equations. The translation equations include driving force terms, friction force terms, and gravity terms. The rotation equations include torque terms and inertial torque terms.

[0039] The six-degree-of-freedom kinematic model is constructed as a state-space expression. The state vector of the state-space expression includes spatial coordinates, movement speed, joint angles, and angular velocity. The control vector of the state-space expression includes driving force and triaxial torque.

[0040] The state-space expression is linearized, and the partial derivatives of the system state equation with respect to the state vector and control vector are calculated to construct a linearized prediction model.

[0041] A trajectory tracking prediction cost function is constructed, which includes a tracking error term, a control quantity penalty term, and a control increment penalty term. The penalty terms are weighted and combined using a weight matrix.

[0042] State constraints are constructed, including spatial coordinate constraints and joint angle constraints; control constraints are constructed, including driving force constraints and torque constraints; and control increment constraints are constructed, including driving force increment constraints and torque increment constraints.

[0043] The trajectory tracking prediction cost function is transformed into the standard form of a quadratic programming problem. The quadratic form matrix and the coefficients of the linear terms are calculated, and the inequality constraint matrix and the equality constraint matrix are constructed.

[0044] The quadratic programming problem is solved using the effective set method, which gradually approaches the optimal solution by identifying effective constraints and solving subproblems.

[0045] Based on the optimization solution results, control quantities are mapped, and the total driving force is distributed to each drive motor through the driving force distribution matrix, and the three-axis torque is distributed to each joint actuator through the torque distribution matrix.

[0046] The output of the drive motor is limited, the output torque is limited according to the rated power of the motor, and the rotation angle is limited according to the maximum rotation speed of the joint, so as to generate the final gripping control command.

[0047] Preferably, the feature extraction processing of the real-time image data based on the feature fusion network to obtain workpiece surface feature data includes: performing grayscale processing on the real-time image data acquired by the front-view camera, side-view camera, and top-view camera respectively to obtain single-channel grayscale images; performing Gaussian filtering to denoise each grayscale image, calculating the local mean and variance through a sliding window, and adjusting the image contrast; using a scale-invariant feature transform algorithm to extract local feature points of each image, establishing the correspondence between images from different perspectives through feature point matching; transforming the coordinates of the matched feature points to a unified world coordinate system, and fusing the feature point information from multiple perspectives through a weighted average method to generate workpiece surface feature data containing depth information.

[0048] Preferably, constructing the particle filtering framework includes: initializing a particle set with a uniform distribution, where each particle contains the position coordinates and rotation angle of the workpiece in three-dimensional space; calculating the end effector pose corresponding to the particle using the forward kinematics of the robotic arm; projecting the particle pose onto the image plane based on the camera extrinsic parameter matrix to generate predicted feature point coordinates; calculating the pixel error between the predicted feature point coordinates and the actual image feature point coordinates; constructing an observation likelihood function based on the sum of squared errors; obtaining the weight value of each particle through normalization processing; retaining particles with weight values ​​greater than a preset threshold; and removing particles with weight values ​​less than a preset threshold.

[0049] Preferably, the state transition cost includes: calculating the Euclidean distance between the current node and the next node as the path length cost; calculating the change in rotation angle of each joint required for the robotic arm to move from the current node to the next node as the joint motion cost; calculating the minimum distance to a fixed obstacle in the path, and increasing the penalty cost if it is less than the safe distance; and generating the total state transition cost by linearly combining the path length cost, joint motion cost, and penalty cost.

[0050] Preferably, the present invention further includes a workpiece gripping system based on visual feedback, used to implement the above-described workpiece gripping method based on visual feedback, the system comprising:

[0051] The first module is used to acquire real-time image data of the target workpiece through a multi-view visual perception device, which includes a front-view camera, a side-view camera, and a top-view camera; to perform feature extraction processing on the real-time image data based on a feature fusion network to obtain workpiece surface feature data; and to input the workpiece surface feature data into a pre-trained pose estimation model, which adopts a particle filter framework and iteratively updates the workpiece pose based on the observation likelihood function to generate workpiece pose prediction parameters.

[0052] The second module is used to construct a multi-objective path planning model based on the workpiece pose prediction parameters. The multi-objective path planning model aims to minimize the gripping path length and the joint motion amplitude. It uses a dynamic programming algorithm to globally optimize the gripping path. The dynamic programming algorithm introduces state transition costs and heuristic evaluation functions to improve path search efficiency. Based on the multi-objective path planning model, it outputs the optimal gripping path data.

[0053] The third module is used to establish a hierarchical grasping control model based on the optimal grasping path data. The hierarchical grasping control model includes a strategy layer, an adjustment layer, and an execution layer. The strategy layer performs global grasping planning based on the workpiece pose prediction parameters, the adjustment layer performs local path correction based on the optimal grasping path data, and the execution layer implements trajectory tracking of the robotic arm based on a visual servo control algorithm. The hierarchical grasping control model outputs grasping control commands to achieve visual feedback grasping control of the target workpiece.

[0054] Compared with the prior art, the beneficial effects of the present invention are:

[0055] Employing a multi-view visual perception device encompassing front, side, and top views, this system comprehensively acquires 3D image data of the workpiece. Combined with a feature fusion network, it extracts and fuses features from multiple perspectives, effectively compensating for the incomplete information provided by single-view images. Through processing techniques such as grayscale conversion, Gaussian filtering, and scale-invariant feature transformation, it achieves accurate matching and unified coordinate system transformation of feature points from different perspectives. The generated workpiece surface feature data includes complete edge contours, surface textures, key feature points, and depth information, providing rich input for subsequent pose estimation and significantly improving the completeness and accuracy of feature extraction in complex environments.

[0056] The pose estimation model employs a particle filter framework and introduces a multi-dimensional observation likelihood function. Through comprehensive evaluation of contour matching, texture alignment, feature point coincidence, and occlusion exclusion terms, it achieves dynamic iterative updates of the workpiece pose. Compared to traditional single feature matching methods, this model effectively integrates multi-source information, resists interference from background occlusion and illumination changes, and retains high-weight particles through a resampling mechanism, improving the robustness and convergence speed of pose prediction. This makes the predicted parameters (such as particle mean coordinates and covariance matrix) closer to the actual workpiece state, providing accurate basic data for path planning.

[0057] The multi-objective path planning model optimizes the path length and joint motion amplitude by combining dynamic programming with state transition costs and a heuristic evaluation function. Through adaptive time-varying weight coefficients and influencing factors, it achieves efficient global path optimization. Compared to single-objective planning, this model ensures path efficiency while preventing the robotic arm's joint motion from exceeding limits, reducing joint wear and energy consumption. Furthermore, it uses the Pareto front solution set to select the optimal path and performs spline smoothing, making the path more closely match the robotic arm's kinematic characteristics and improving the stability of the grasping process.

[0058] The hierarchical grasping control model achieves an organic combination of global planning and local correction through the collaborative work of the strategy layer, adjustment layer, and execution layer. The strategy layer performs global multi-objective optimization based on pose parameters, ensuring the overall rationality of the path; the adjustment layer, through local window adaptive adjustment and Kalman filter tracking, can quickly respond to sudden obstacles and path deviations; and the execution layer, based on visual servo control and quadratic programming, achieves high-precision trajectory tracking. This hierarchical structure reduces the coupling between modules, improves the system's adaptability to dynamic environments, and enables the robotic arm to maintain high grasping accuracy and efficiency even in complex scenarios, effectively meeting the high-precision and high-reliability requirements for workpiece grasping in industrial automation. Attached Figure Description

[0059] Figure 1 This is a schematic diagram illustrating the working principle of the visual feedback-based workpiece grasping method described in this invention.

[0060] Figure 2 A flowchart for iterative updates of the pose estimation model;

[0061] Figure 3 A flowchart for optimizing a multi-objective path planning model;

[0062] Figure 4 A flowchart for global capture planning at the strategy layer;

[0063] Figure 5 This is a flowchart for adjusting the local path correction of the adjustment layer. Detailed Implementation

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] Please see Figures 1-5 The present invention relates to a workpiece gripping method based on visual feedback, the specific implementation steps of which are as follows:

[0066] Real-time image data of the target workpiece is acquired through a multi-view visual perception device, which includes a front-view camera, a side-view camera, and a top-view camera. Feature extraction processing is performed on the real-time image data based on a feature fusion network to obtain workpiece surface feature data. The workpiece surface feature data is then input into a pre-trained pose estimation model, which uses a particle filter framework and iteratively updates the workpiece pose based on the observation likelihood function to generate workpiece pose prediction parameters.

[0067] A multi-objective path planning model is constructed based on the workpiece pose prediction parameters. The multi-objective path planning model aims to minimize the gripping path length and the joint motion amplitude. A dynamic programming algorithm is used to globally optimize the gripping path. The dynamic programming algorithm introduces state transition costs and heuristic evaluation functions. The optimal gripping path data is output based on the multi-objective path planning model.

[0068] A hierarchical grasping control model is established based on the optimal grasping path data. The hierarchical grasping control model includes a strategy layer, an adjustment layer, and an execution layer. The strategy layer performs global grasping planning based on the workpiece pose prediction parameters, the adjustment layer performs local path correction based on the optimal grasping path data, and the execution layer realizes the trajectory tracking of the robotic arm based on a visual servo control algorithm. The hierarchical grasping control model outputs grasping control commands to achieve visual feedback grasping control of the target workpiece.

[0069] Example 1:

[0070] In the process of inputting workpiece surface feature data into a pre-trained pose estimation model, real-time image data needs to be acquired. This real-time image data contains multiple aspects of information, including workpiece edge contour information, which outlines the external shape of the workpiece; surface texture information, which reflects the texture features of the workpiece surface; coordinates of key feature points, which are important for determining the position and orientation of the workpiece; and background occlusion information, used to understand the occlusion of the workpiece by the surrounding environment. A state space is constructed based on this real-time image data. Simultaneously, a motion space is constructed based on the translational motion, rotational angle, and gripper opening / closing degree that the robotic arm's end effector can execute.

[0071] A multi-dimensional observation likelihood function is constructed based on the established state space and action space. This function contains several important terms. The contour matching term is obtained by calculating the shape similarity between the actual and predicted contours of the workpiece. This shape similarity calculation allows the model to better determine the degree of fit between the predicted and actual contours. The texture alignment term is obtained by calculating the grayscale correlation between the workpiece surface texture and the template texture. This grayscale correlation calculation reflects the similarity between textures. The feature point coincidence term is obtained by calculating the positional deviation between the actual and predicted feature point coordinates. The magnitude of the positional deviation reflects the accuracy of the feature point prediction. The occlusion exclusion term is obtained by calculating the pixel separation between the background region and the workpiece region. Pixel separation helps the model exclude the influence of background occlusion.

[0072] A particle filtering framework is constructed, comprising several components. The initial particle set contains multiple workpiece pose hypothesis samples, representing various assumptions about the possible poses of the workpiece. The state transition function predicts and updates the particle poses using the robotic arm's kinematic model, which can predict the particle's pose at the next moment based on the current state. The observation likelihood calculation module evaluates the particle weights based on the previously constructed multi-dimensional observation likelihood function, determining which particles are closer to the workpiece's true pose.

[0073] The initial particle set is updated using a resampling method. Specifically, a roulette wheel selection process is used to retain high-weight particles, which are more likely to represent the true pose of the workpiece. Simultaneously, low-weight particles are removed and new particles are added. This continuous optimization of the particle set ensures it better reflects the actual situation of the workpiece. Based on the updated particle set, the mean and covariance of the workpiece pose are calculated. The mean represents the approximate position of the workpiece, while the covariance reflects the uncertainty of the pose.

[0074] The particle filter framework outputs pose prediction parameters, which include the particle mean coordinates, which give the average position of the workpiece pose; the covariance matrix, which describes the distribution of the pose; the confidence of key feature point matching, which reflects the reliability of key feature point matching; and the probability of occlusion region exclusion, which reflects the possibility of handling occluded regions.

[0075] When constructing the particle filtering framework, a uniformly distributed particle set is used for initialization. Each particle contains the workpiece's position coordinates and rotation angle in 3D space, thus uniformly distributing assumptions about the workpiece's pose in 3D space. The end effector pose corresponding to the particle is calculated using the forward kinematics of the robotic arm. The forward kinematics of the robotic arm can determine the position and orientation of the end effector based on the particle's pose. The particle pose is projected onto the image plane based on the camera extrinsic parameter matrix to generate predicted feature point coordinates. The camera extrinsic parameter matrix enables the transformation from 3D space to the 2D image plane. The pixel error between the predicted feature point coordinates and the actual image feature point coordinates is calculated. An observation likelihood function is constructed based on the sum of squared errors, which measures the degree of difference between the predicted and actual feature points. The weight values ​​of each particle are obtained through normalization. Normalization makes the weight values ​​comparable. Particles with weight values ​​greater than a preset threshold are retained, while particles with weight values ​​less than the preset threshold are removed. This filters out more reliable particles and improves the accuracy of pose estimation.

[0076] Example 2:

[0077] When constructing a multi-objective path planning model based on workpiece pose prediction parameters, a multi-objective function for path planning is constructed, which includes two main optimization objective functions. The path length optimization objective function is obtained by dividing the path into segments, calculating the ratio of the straight-line distance of each segment to the corresponding movement speed, and then summing these ratios. This calculation method comprehensively considers the influence of path length and movement speed on path planning. The joint motion optimization objective function is calculated by weighted summation of the rotation angle, translational motion, and acceleration change of each joint. This weighted summation allows for a reasonable combination based on the importance of different motion parameters.

[0078] Based on the aforementioned multi-objective function, path planning constraints are constructed to ensure the safety and feasibility of the robotic arm during its movement. Velocity constraints limit the range of movement speed of the robotic arm's end effector, ensuring the arm's speed remains within a reasonable range and avoiding problems caused by excessive speed or slowness. Angle constraints limit the range of rotation angles of each joint, ensuring joint movement remains within a safe angle range and preventing damage from excessive joint rotation. Acceleration constraints limit the range of acceleration values ​​for the robotic arm; reasonable acceleration control makes the robotic arm's movement smoother. Obstacle constraints ensure a safe distance between the robotic arm and surrounding fixed obstacles, preventing collisions during movement.

[0079] The grasping path is encoded using a node sequence, with each node containing spatial coordinates and robotic arm joint angle information. This encoding method can accurately describe every position and robotic arm posture on the grasping path. State transition probabilities are constructed based on state transition costs and heuristic evaluation values, and the next feasible node is determined using these probabilities. The heuristic evaluation values ​​are obtained through a weighted calculation of the straight-line distance between nodes and the joint motion amplitude, which comprehensively considers the influence of distance and motion amplitude on node selection.

[0080] An adaptive state transition cost mechanism is introduced, employing a time-varying weight coefficient that linearly decreases with the number of planning iterations. This time-varying weight coefficient dynamically adjusts the state transition cost. Initially, a larger weight coefficient significantly impacts path selection; as iterations increase, the coefficient gradually decreases, achieving dynamic adjustment. Simultaneously, a dynamic heuristic evaluation mechanism is introduced, using a time-varying influence factor that linearly increases with the number of iterations. This factor adjusts the weight of the heuristic evaluation value in the state transition probability. During planning, the weight of the heuristic evaluation value gradually increases with iterations, making path planning more reliant on heuristic information and improving path search efficiency.

[0081] Iterative optimization is performed based on an adaptive state transition cost mechanism and a dynamic heuristic evaluation mechanism. In each iteration, the length cost and joint motion cost of the generated path are evaluated, and non-dominated solutions are added to the Pareto front solution set. The Pareto front solution set contains solutions that cannot be improved simultaneously on both the path length and joint motion amplitude objectives; these solutions are all valid candidate solutions.

[0082] The optimal path is selected from the Pareto front solution set, satisfying the length-motion tradeoff. This optimal solution achieves a good balance between path length and joint motion amplitude, minimizing both path length and joint motion amplitude. The optimal path is then smoothed using spline curves. This smoothing process makes the path smoother, reducing impact and vibration during the robotic arm's movement. An optimal node sequence and corresponding robotic arm motion parameters are generated, which will guide the actual movement of the robotic arm.

[0083] The construction of state transition costs involves multiple calculations. First, the Euclidean distance between the current node and the next node is calculated and used as the path length cost. Euclidean distance directly reflects the distance between nodes. Next, the change in rotation angle of each joint required for the robotic arm to move from the current node to the next node is calculated as the joint motion cost. The magnitude of the rotation angle change reflects the amplitude of joint movement. Then, the minimum distance to fixed obstacles in the path is calculated. If this minimum distance is less than the safe distance, a penalty cost is added. The penalty cost incentivizes path planning to avoid obstacles. Finally, the total state transition cost is generated by linearly combining the path length cost, joint motion cost, and penalty cost. This linear combination integrates all cost factors to form a comprehensive state transition cost assessment.

[0084] Example 3:

[0085] When the strategy layer performs global grasping planning based on workpiece pose prediction parameters, it uses parametric curves to describe the grasping path, representing the grasping path as a function of path parameters. The path parameters range from 0 to 1, and the grasping path includes X-axis, Y-axis, and Z-axis coordinate components. This parametric description method can accurately represent the direction of the grasping path in three-dimensional space through changes in the path parameters.

[0086] The grasping path is described using a quintic Bézier curve. Specifically, the spatial coordinates of the grasping path are calculated by summing the products of the control vertex coordinates and the Bézier basis functions. The Bézier basis functions are calculated as power functions of the combination number and the path parameters. Using a quintic Bézier curve, a grasping path shape that meets the requirements can be fitted relatively flexibly. The direction and shape of the path can be changed by adjusting the position of the control vertices.

[0087] Constraints are constructed for the grasping path to ensure its feasibility and safety. Position constraints limit the coordinate range of the grasping path, ensuring it lies within a spatial region accessible to the robotic arm and meeting operational requirements. Velocity constraints limit the range of the first derivative of the grasping path; the first derivative reflects the rate of change of the path, and a reasonable velocity range ensures the robotic arm's movement speed meets requirements. Acceleration constraints limit the range of the second derivative of the grasping path; the second derivative is related to acceleration, and limiting the acceleration range ensures smooth robotic arm movement and reduces impact. Curvature constraints limit the range of curvature variation in the grasping path; curvature reflects the degree of path bending, and reasonable control of curvature variation prevents excessively sharp bends, allowing the robotic arm to move smoothly along the path.

[0088] A global multi-objective optimization function is constructed, comprising a total path length term, a curvature integral term, a curvature change rate integral term, and an obstacle distance metric term. The total path length term measures the length of the path, the curvature integral term reflects the total curvature of the path, the curvature change rate integral term reflects the rate of change of the path's curvature, and the obstacle distance metric term assesses the distance between the path and obstacles. Weighting coefficients are used to weight and combine the terms in the global multi-objective optimization function, assigning different weights based on actual needs and the importance of each objective. This allows the optimization function to comprehensively consider multiple factors and achieve multi-objective optimization of the grasping path.

[0089] The captured path interval is discretized into multiple path segments, and the global multi-objective optimization function is discretized to construct a global discretized objective function. The global discretized objective function includes path segment length, path segment curvature, curvature change, and minimum distance to obstacles. By discretizing, the continuous path optimization problem is transformed into a discrete problem for processing, which facilitates the use of optimization algorithms for solution.

[0090] A gradient descent method is used to iteratively optimize the globally discretized objective function. Specifically, the gradient of the objective function with respect to the control vertices is calculated, and then the position coordinates of the control vertices are updated along the negative gradient direction. By continuously iterating and updating the positions of the control vertices, the value of the objective function gradually decreases, thereby finding a better grasping path.

[0091] The optimized grasping path is smoothed using cubic spline interpolation. By maintaining the continuity of the position derivative, velocity derivative, and acceleration derivative at the interpolation endpoints, a smooth and continuous grasping path is generated. This smoothing process makes the path smoother, preventing vibrations or impacts to the robotic arm during movement caused by an uneven path. Based on the smooth and continuous grasping path, a node sequence and corresponding robotic arm movement speed are generated. The node sequence clearly defines the positions the robotic arm needs to traverse during grasping, while the movement speed parameters provide the basis for the robotic arm's motion control.

[0092] Throughout the entire global grasping planning process, from the parameterized description of the grasping path to the application of Bézier curves, and then to the construction of various constraints and the setting of optimization functions, each step is interconnected and works together to generate the optimal grasping path. The use of parameterized curves and quintic Bézier curves provides a flexible way to describe the grasping path, the setting of constraints ensures the feasibility and safety of the path, the construction of multi-objective optimization functions comprehensively considers multiple important factors such as path length, curvature, and obstacle distance, the iterative optimization process of the gradient descent method continuously improves the path, and cubic spline interpolation smoothing makes the path more suitable for the movement of the robotic arm.

[0093] By strictly following the above steps, global grasping planning can be completed based on workpiece pose prediction parameters, generating a grasping path and corresponding robotic arm motion parameters that meet the requirements. During implementation, each step requires precise processing to ensure accuracy, enabling the strategy layer to effectively perform global grasping planning and providing reliable path guidance for subsequent robotic arm grasping operations. The entire process emphasizes details and the logical relationships between steps to achieve efficient and accurate global grasping planning.

[0094] Example 4:

[0095] When the adjustment layer performs local path correction based on the optimal grasping path data, it constructs a local planning window based on the robot arm's current position, current joint angles, and current movement speed. Here, the current position is the robot arm's actual coordinates in space, the current joint angles reflect the posture of each joint, and the current movement speed is the motion rate of the robot arm's end effector. The size of the local planning window is adaptively adjusted using a speed correlation coefficient, establishing a positive correlation between the window size and the current movement speed. For example, when the robot arm moves at a higher speed, the size of the local planning window will increase accordingly to plan a longer path in advance; conversely, when the movement speed is lower, the size of the local planning window will decrease, focusing more on path correction in the immediate vicinity.

[0096] A local environment model is constructed using multi-view visual data. This data comes from images captured by front-view, side-view, and top-view cameras. Coordinate transformation is performed on the image data to obtain positional data in a local coordinate system. This transformation converts image data from different cameras to a unified local coordinate system, ensuring data consistency and comparability. The occupancy probability of a raster map is updated based on the positional data. The raster map divides the local environment into multiple small raster cells, each with an occupancy probability value representing the likelihood of an object occupying that location. The occupancy probability of each raster cell is calculated using a probability accumulation method. As new visual data is continuously input, the occupancy probability of the raster cells is updated according to the probability accumulation rule, gradually building an accurate local environment model.

[0097] The Kalman filter algorithm is used to track dynamic obstacles. The Kalman filter is a commonly used state estimation method that predicts the obstacle's state vector through a state prediction equation. This state vector includes information such as the obstacle's position and velocity. The predicted state is updated based on measurement data obtained from multi-view camera observations of the obstacle. By combining the predicted state with the measurement data, the precise position and motion information of the obstacle are obtained. For example, when a dynamic obstacle moves in a scene, the Kalman filter algorithm predicts its next position based on its previous motion state, and then corrects the prediction using camera observation data, thereby accurately tracking the obstacle's trajectory.

[0098] A path correction model is constructed. The kinematic equations of the robotic arm are used as state equations, which include spatial coordinates, movement velocity, and joint angles. These state variables describe the motion state of the robotic arm. State constraints and dynamic constraints are constructed. State constraints limit the range of values ​​for spatial coordinates and movement velocity to ensure that the motion state of the robotic arm is within a safe and feasible range. Dynamic constraints limit the range of values ​​for angular velocity, angular acceleration, and the rate of change of joint angles to ensure that the motion of the robotic arm conforms to dynamic characteristics and avoids excessive acceleration or velocity changes.

[0099] A multi-objective cost function is then constructed, comprising a reference trajectory tracking term, an obstacle avoidance term, a trajectory smoothing term, and an energy consumption term. The reference trajectory tracking term measures the deviation between the corrected path and the optimal grasping path, aiming to approximate the reference trajectory as closely as possible. The obstacle avoidance term assesses the distance between the path and obstacles, ensuring the path avoids obstacles. The trajectory smoothing term ensures the smoothness of the path, reducing vibration during the robotic arm's movement. The energy consumption term considers the energy consumption of the robotic arm's movement, pursuing energy optimization. Weighting coefficients are used to weight and combine the terms in the multi-objective cost function, determining different weights based on actual needs and the importance of each objective, enabling the cost function to comprehensively reflect multiple optimization objectives.

[0100] The Lagrange multiplier method is used to optimize the multi-objective cost function. The Lagrange multiplier method is an effective method for solving constrained optimization problems. By constructing a Lagrange function and introducing constraints, the constrained optimization problem is transformed into an unconstrained optimization problem. The optimal control variables, including the control parameters of each joint of the robotic arm, are obtained by solving the partial derivative equations and are used to guide the correction of the robotic arm's motion path.

[0101] Throughout the local path correction process, each step works in close coordination. Constructing a local planning window allows the system to dynamically adjust the planning range based on the robot arm's current motion state. A local environment model built from multi-view visual data provides accurate environmental information for path correction. The Kalman filter algorithm tracks dynamic obstacles, ensuring the system can respond promptly to dynamic changes in the environment. The path correction model and multi-objective cost function provide a mathematical model and optimization objectives for path correction. Finally, the optimization solution using the Lagrange multiplier method yields a feasible path correction scheme.

[0102] For example, when a robotic arm is moving along the optimal grasping path, it suddenly detects a dynamic obstacle (such as another moving robotic arm). At this time, the local planning window will adaptively adjust its size according to the current moving speed. The multi-view camera acquires visual data of the obstacle, updates the occupancy probability of the grid map through coordinate transformation, and the Kalman filter algorithm tracks the trajectory of the obstacle. The path correction model constructs state equations and constraints based on the current state of the robotic arm and obstacle information. The multi-objective cost function comprehensively considers factors such as reference trajectory tracking, obstacle avoidance, trajectory smoothing, and energy consumption. Finally, the optimal control quantity is obtained by solving the Lagrange multiplier method, so that the robotic arm can avoid the obstacle in time while staying as close as possible to the original optimal grasping path.

[0103] Through this implementation, the adjustment layer can effectively correct local paths based on optimal grasping path data, enabling the robotic arm to safely and reliably complete grasping tasks in complex working environments and cope with various sudden environmental changes and obstacle interference. The entire process focuses on scene changes and the robotic arm's motion characteristics in practical applications, achieving precise local path correction through a series of processing steps and algorithms.

[0104] Example 5:

[0105] When the execution layer implements trajectory tracking of the robotic arm based on the visual servo control algorithm, a six-degree-of-freedom kinematic model of the robotic arm needs to be established. This model includes translation equations and rotation equations. The translation equations include driving force terms, friction terms, and gravity terms. The driving force term is the power source that causes the robotic arm's translational motion, the friction term reflects the frictional resistance encountered during the motion, and the gravity term is due to the influence of the robotic arm's own weight. The rotation equations include torque terms and inertial torque terms. The torque term is the torque that causes the robotic arm's joints to rotate, and the inertial torque term is related to the robotic arm's moment of inertia and angular acceleration, describing the inertial characteristics of the robotic arm's rotational motion.

[0106] The six-degree-of-freedom kinematic model is constructed as a state-space expression. The state vectors of the state-space expression include spatial coordinates, translational velocity, joint angles, and angular velocities. These state vectors comprehensively describe the motion state of the robotic arm. Spatial coordinates represent the position of the robotic arm's end effector in three-dimensional space, translational velocity is the motion rate of the end effector, joint angles reflect the posture of each joint, and angular velocity is the speed of joint rotation. The control vectors of the state-space expression include driving force and three-axis torque. The driving force is used to control the translational motion of the robotic arm, and the three-axis torque is used to control the rotational motion of the joints.

[0107] The state-space expression is linearized by calculating the partial derivatives of the system state equations with respect to the state vector and control vector. A linearized prediction model is then constructed using these partial derivatives. Linearization is performed to approximate a complex nonlinear system as a linear system, facilitating subsequent analysis and control.

[0108] A trajectory tracking prediction cost function is constructed, which includes a tracking error term, a control input penalty term, and a control increment penalty term. The tracking error term measures the deviation between the actual trajectory of the robotic arm and the desired trajectory. The control input penalty term penalizes excessively large control inputs, while the control increment penalty term restricts drastic changes in the control input. The penalty terms are weighted and combined using a weight matrix, assigning appropriate weights to different penalty terms according to actual control requirements to balance the importance of each objective.

[0109] Then, various constraints are constructed. State constraints include spatial coordinate constraints and joint angle constraints. Spatial coordinate constraints limit the spatial range that the robotic arm's end effector can reach, while joint angle constraints ensure that the rotation angle of each joint is within a safe range. Control constraints include driving force constraints and torque constraints. Driving force constraints specify the range of values ​​for the driving force, while torque constraints limit the magnitude of the torque on each axis. Control increment constraints include driving force increment constraints and torque increment constraints, used to limit the variation range of driving force and torque to avoid abrupt changes in control quantities.

[0110] The trajectory tracking prediction cost function is transformed into the standard form of a quadratic programming problem. The quadratic form matrix and the coefficients of the linear terms are calculated, and inequality and equality constraint matrices are constructed. The standard form of the quadratic programming problem facilitates solution using specific optimization algorithms, and matrix operations transform the problem into a mathematically tractable form.

[0111] The efficient set method is used to solve quadratic programming problems by identifying effective constraints and solving subproblems, gradually approaching the optimal solution. The efficient set method is a commonly used algorithm for solving quadratic programming problems, capable of finding the optimal control quantity while considering constraints.

[0112] Based on the optimization results, control quantities are mapped. A driving force distribution matrix allocates the total driving force to each drive motor, ensuring a reasonable distribution based on the robot arm's structure and kinematics. A torque distribution matrix distributes the three-axis torque to each joint actuator, ensuring that the three-axis torque is correctly applied to each joint, thus achieving control of joint rotation.

[0113] The output of the drive motor is limited, with the output torque limited according to the motor's rated power to prevent damage caused by overload. The rotation angle is limited according to the joint's maximum rotation speed to ensure that the joint's rotation speed is within a safe range, preventing vibration or damage to the robotic arm due to excessive speed. After the limiting process, the final grasping control command is generated to control the actual movement of the robotic arm.

[0114] Taking a scenario where a six-axis robotic arm grasps a workpiece as an example, when the robotic arm needs to move from its initial position to its target position to grasp the workpiece, the execution layer first establishes a six-degree-of-freedom kinematic model, considering the driving force, friction, and gravity in the translation equations, and the torque and inertial moment in the rotation equations. After constructing the model as a state-space expression, it is linearized to obtain a linearized prediction model. A trajectory tracking prediction cost function is constructed, where the tracking error term focuses on whether the robotic arm accurately tracks the preset grasping path, the control quantity penalty term avoids excessive driving force and torque input, and the control increment penalty term prevents drastic changes in the control quantity.

[0115] Suppose that during the movement, the actual trajectory of the robotic arm deviates from the desired trajectory, for example, the end effector deviates from the preset path by 5 mm in the X-axis direction. In this case, the value of the tracking error term will increase. Simultaneously, if the current control variable causes the rotation angle of a joint to approach its maximum limit, the control constraint condition will be triggered. A quadratic programming problem is solved using the effective set method to obtain the adjusted control variable, such as increasing the driving force in the X-axis direction and adjusting the torque of the relevant joints, so that the robotic arm returns to the desired trajectory. The driving force allocation matrix distributes the total driving force to the corresponding drive motors, and the torque allocation matrix distributes the three-axis torque to each joint driver. Finally, the motor output is limited to ensure that the motors and joints operate within safe ranges, generating the final grasping control command so that the robotic arm accurately reaches the target position to grasp the workpiece.

[0116] From establishing the kinematic model to generating control commands, every step revolves around the trajectory tracking control of the robotic arm. Through precise modeling, linearization, cost function construction, constraint setting, and optimization solutions of the robotic arm's motion state, accurate control of its movement is achieved. In practical applications, whether dealing with simple linear motion or complex curved trajectories, this implementation method ensures that the robotic arm moves accurately along the preset trajectory to complete the workpiece grasping task through a series of calculations and control steps. Simultaneously, various constraints and amplitude limiting measures ensure the safety and reliability of the robotic arm's movement, preventing equipment damage or task failure due to improper control.

[0117] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0118] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A workpiece grasping method based on visual feedback, characterized by, Comprise: Obtain real-time image data of the target workpiece through a multi-view visual perception device, the multi-view visual perception device comprising a front-view camera, a side-view camera and a top-view camera; Perform feature extraction processing on the real-time image data based on a feature fusion network to obtain workpiece surface feature data; input the workpiece surface feature data into a pre-trained pose estimation model, the pose estimation model adopting a particle filtering framework, iteratively updating the workpiece pose based on an observation likelihood function to generate workpiece pose prediction parameters; Construct a multi-target path planning model according to the workpiece pose prediction parameters, the multi-target path planning model taking the shortest grabbing path length and the smallest joint motion amplitude as optimization objectives, and globally optimizing the grabbing path using a dynamic programming algorithm, wherein the dynamic programming algorithm introduces a state transition cost and a heuristic evaluation function; output optimal grabbing path data based on the multi-target path planning model; Establish a hierarchical grabbing control model according to the optimal grabbing path data, the hierarchical grabbing control model comprising a strategy layer, an adjustment layer and an execution layer, wherein the strategy layer performs global grabbing planning based on the workpiece pose prediction parameters, the adjustment layer performs local path correction based on the optimal grabbing path data, and the execution layer realizes trajectory tracking of the mechanical arm based on a visual servo control algorithm; output grabbing control instructions through the hierarchical grabbing control model to realize visual feedback grabbing control of the target workpiece; Input the workpiece surface feature data into a pre-trained pose estimation model, the pose estimation model adopting a particle filtering framework, iteratively updating the workpiece pose based on an observation likelihood function to generate workpiece pose prediction parameters comprise: Obtain real-time image data, the real-time image data comprising workpiece edge contour information, surface texture information, key feature point coordinates and background environment occlusion information; construct a state space based on the real-time image data, and construct an action space based on the translational motion amount, rotational angle amount and gripper opening degree executable by the end effector of the mechanical arm; Construct a multi-dimensional observation likelihood function based on the state space and the action space, the multi-dimensional observation likelihood function comprising a contour matching term, a texture alignment term, a feature point coincidence term and an occlusion exclusion term, wherein the contour matching term is calculated by the shape similarity of the actual contour and the predicted contour of the workpiece, the texture alignment term is calculated by the gray correlation of the workpiece surface texture and the template texture, the feature point coincidence term is calculated by the positional deviation of the actual feature point coordinates and the predicted feature point coordinates, and the occlusion exclusion term is calculated by the pixel separation degree of the background area and the workpiece area; Construct a particle filtering framework, the particle filtering framework comprising an initial particle set, a state transition function and an observation likelihood calculation module, the initial particle set containing multiple workpiece pose hypothesis samples, the state transition function predicting and updating the particle pose through the kinematics model of the mechanical arm, and the observation likelihood calculation module evaluating the particle weight based on the multi-dimensional observation likelihood function; The initial particle set is updated by a resampling method, high-weight particles are selected by roulette selection, low-weight particles are removed and new particles are supplemented, and the mean value and covariance of the workpiece pose are calculated based on the updated particle set; the pose prediction parameters are output based on the particle filtering framework, and the pose prediction parameters include particle mean coordinates, covariance matrix, key feature point matching confidence and occlusion area exclusion probability; A multi-objective path planning model is constructed based on the workpiece pose prediction parameters, the multi-objective path planning model takes the shortest grasping path length and the smallest joint motion amplitude as optimization objectives, a dynamic programming algorithm is used for global optimization of the grasping path, and optimal grasping path data is output based on the multi-objective path planning model, including: A multi-objective function of path planning is constructed, the multi-objective function includes a path length optimization objective function and a joint motion optimization objective function, wherein the path length optimization objective function is calculated by summing the ratio of path segmented straight line distance to corresponding moving speed, and the joint motion optimization objective function is calculated by weighted sum of joint rotation angle, translation motion amount and acceleration change amount; A path planning constraint condition is constructed based on the multi-objective function, the path planning constraint condition includes a speed constraint, an angle constraint, an acceleration constraint and an obstacle constraint, the speed constraint is used to limit the moving speed range of the end effector of the robot arm, the angle constraint is used to limit the change range of the rotation angle of each joint, the acceleration constraint is used to limit the value range of the motion acceleration of the robot arm, and the obstacle constraint is used to ensure the safety distance between the robot arm and the surrounding fixed obstacles; A node sequence is used to encode the grasping path, each node contains spatial coordinates and robot arm joint angle information, a state transition probability is constructed based on state transition cost and heuristic evaluation value, and the next feasible node is determined through the state transition probability; wherein the heuristic evaluation value is calculated by weighting the straight line distance between nodes and the joint motion amplitude; An adaptive state transition cost mechanism is introduced, the adaptive state transition cost mechanism uses a time-varying weight coefficient, the time-varying weight coefficient changes linearly with the increase of the planning iteration number, and the state transition cost is dynamically adjusted through the time-varying weight coefficient; A dynamic heuristic evaluation mechanism is introduced, the dynamic heuristic evaluation mechanism uses a time-varying influence factor, the time-varying influence factor changes linearly with the increase of the planning iteration number, and the weight of the heuristic evaluation value in the state transition probability is adjusted through the time-varying influence factor; based on the adaptive state transition cost mechanism and the dynamic heuristic evaluation mechanism, iterative optimization is performed, the path generated by each iteration is evaluated in terms of length cost and joint motion cost, and non-dominated solutions are added to the Pareto front solution set; An optimal solution that meets the length-motion trade-off is selected from the Pareto front solution set as the optimal path, and the optimal path is subjected to spline curve smoothing processing to generate an optimal node sequence and corresponding robot arm motion parameters.

2. The workpiece grasping method based on visual feedback according to claim 1, characterized by, The strategy layer performs global grasping planning based on the workpiece pose prediction parameters, including: The grabbing path is described by using a parameterized curve, and the grabbing path is represented as a function of path parameters, the value range of the path parameters is 0 to 1, and the grabbing path includes an X-axis coordinate component, a Y-axis coordinate component and a Z-axis coordinate component; The grabbing path is described based on a quintic Bezier curve, and the spatial coordinates of the grabbing path are calculated by controlling the summation of the product of vertex coordinates and a Bezier base function, the Bezier base function is calculated by combining a number and a power function of the path parameter; A grabbing path constraint condition is constructed, the grabbing path constraint condition includes a position constraint, a velocity constraint, an acceleration constraint and a curvature constraint, the position constraint is used to limit the coordinate range of the grabbing path, the velocity constraint is used to limit the first derivative range of the grabbing path, the acceleration constraint is used to limit the second derivative range of the grabbing path, and the curvature constraint is used to limit the curvature change range of the grabbing path; A global multi-objective optimization function is constructed, the global multi-objective optimization function includes a path total length term, a curvature integral term, a curvature change rate integral term and an obstacle distance measure term, and each term in the global multi-objective optimization function is combined by weighting coefficients; The grabbing path interval is discretized into a plurality of path segments, the global multi-objective optimization function is discretized, a global discretized objective function is constructed, and the global discretized objective function includes a path segment length, a path segment curvature, a curvature change amount and a minimum obstacle distance; The global discretized objective function is iteratively optimized by using a gradient descent method, the gradient value of the objective function with respect to the control vertex is calculated, and the position coordinates of the control vertex are updated along the negative gradient direction; The optimized grabbing path is smoothed by using cubic spline interpolation, a smooth and continuous grabbing path is generated by maintaining the continuity of the position derivative, the velocity derivative and the acceleration derivative at the interpolation endpoints, and a node sequence and a corresponding mechanical arm moving speed are generated based on the smooth and continuous grabbing path.

3. The visual feedback-based workpiece grasping method according to claim 1, wherein, The adjustment layer performs local path correction based on the optimal grabbing path data, including: A local planning window is constructed based on the current position, the current joint angle and the current moving speed of the mechanical arm, the size of the local planning window is adaptively adjusted by a velocity correlation coefficient, and the size of the local planning window is positively correlated with the size of the current moving speed; A local environment model is constructed by using multi-view visual data, position data in a local coordinate system is obtained by coordinate transformation of image data, the occupancy probability of a grid map is updated based on the position data, and the occupancy probability value of each grid is calculated by a probability accumulation method; A Kalman filtering algorithm is used to track dynamic obstacles, the state vector of the obstacle is predicted by a state prediction equation, and the accurate position and motion information of the obstacle are obtained by updating the predicted state based on measurement data. A path correction model is constructed, and the kinematic equation of the robotic arm is used as the state equation. The state equation includes spatial coordinates, movement speed and joint angle. State constraints and dynamic constraints are constructed. The state constraints are used to limit the range of values ​​for spatial coordinates and movement speed. The dynamic constraints are used to limit the range of values ​​for angular velocity, angular acceleration and joint angle change rate. A multi-objective cost function is constructed, which includes a reference trajectory tracking term, an obstacle avoidance term, a trajectory smoothing term, and an energy consumption term. The terms in the multi-objective cost function are weighted and combined using weighting coefficients. The Lagrange multiplier method is used to optimize the multi-objective cost function. A Lagrange function is constructed and constraints are introduced. The optimal control quantity is obtained by solving the partial derivative equations.

4. The visual feedback-based workpiece grasping method according to claim 1, wherein, The execution layer uses a visual servo control algorithm to track the trajectory of the robotic arm. It outputs grasping control commands through the hierarchical grasping control model to achieve visual feedback grasping control of the target workpiece, including: A six-degree-of-freedom kinematic model of a robotic arm is established. The six-degree-of-freedom kinematic model includes translation equations and rotation equations. The translation equations include driving force terms, friction force terms, and gravity terms. The rotation equations include torque terms and inertial torque terms. The six-degree-of-freedom kinematic model is constructed as a state-space expression. The state vector of the state-space expression includes spatial coordinates, movement speed, joint angles, and angular velocity. The control vector of the state-space expression includes driving force and triaxial torque. The state-space expression is linearized, and the partial derivatives of the system state equation with respect to the state vector and control vector are calculated to construct a linearized prediction model. A trajectory tracking prediction cost function is constructed, which includes a tracking error term, a control quantity penalty term, and a control increment penalty term. The penalty terms are weighted and combined using a weight matrix. State constraints are constructed, including spatial coordinate constraints and joint angle constraints; control constraints are constructed, including driving force constraints and torque constraints; and control increment constraints are constructed, including driving force increment constraints and torque increment constraints. The trajectory tracking prediction cost function is transformed into the standard form of a quadratic programming problem. The quadratic form matrix and the coefficients of the linear terms are calculated, and the inequality constraint matrix and the equality constraint matrix are constructed. The quadratic programming problem is solved using the effective set method, which gradually approaches the optimal solution by identifying effective constraints and solving subproblems. Based on the optimization solution results, control quantities are mapped, and the total driving force is distributed to each drive motor through the driving force distribution matrix, and the three-axis torque is distributed to each joint actuator through the torque distribution matrix. The output of the drive motor is limited, the output torque is limited according to the rated power of the motor, and the rotation angle is limited according to the maximum rotation speed of the joint, so as to generate the final gripping control command.

5. The visual feedback-based workpiece grasping method according to claim 1, wherein, The feature extraction processing of the real-time image data based on the feature fusion network to obtain workpiece surface feature data includes: grayscale processing of the real-time image data acquired by the front-view camera, side-view camera, and top-view camera to obtain single-channel grayscale images; Gaussian filtering for noise reduction of each grayscale image, calculation of local mean and variance through a sliding window, and adjustment of image contrast; extraction of local feature points of each image using a scale-invariant feature transform algorithm, establishment of correspondence between images from different perspectives through feature point matching; transformation of the coordinates of the matched feature points to a unified world coordinate system, and fusion of multi-view feature point information using a weighted average method to generate workpiece surface feature data containing depth information.

6. The visual feedback-based workpiece grasping method according to claim 1, wherein, The particle filtering framework is constructed as follows: a uniformly distributed particle set is initialized, with each particle containing the position coordinates and rotation angle of the workpiece in three-dimensional space; the end effector pose corresponding to the particle is calculated using the forward kinematics of the robotic arm, and the particle pose is projected onto the image plane based on the camera extrinsic parameter matrix to generate predicted feature point coordinates; the pixel error between the predicted feature point coordinates and the actual image feature point coordinates is calculated, and an observation likelihood function is constructed based on the sum of squared errors; the weight value of each particle is obtained through normalization, and particles with weight values ​​greater than a preset threshold are retained, while particles with weight values ​​less than a preset threshold are removed.

7. The visual feedback-based workpiece grasping method according to claim 1, wherein, The state transition cost is constructed by: calculating the Euclidean distance between the current node and the next node as the path length cost; calculating the change in rotation angle of each joint required for the robotic arm to move from the current node to the next node as the joint motion cost; calculating the minimum distance to a fixed obstacle in the path, and increasing the penalty cost if it is less than the safe distance; and generating the total state transition cost by linearly combining the path length cost, joint motion cost, and penalty cost.

8. A workpiece gripping system based on visual feedback for implementing the workpiece gripping method based on visual feedback according to any one of claims 1 to 7, characterized in that include: The first module is used to acquire real-time image data of the target workpiece through a multi-view visual perception device, which includes a front-view camera, a side-view camera, and a top-view camera. The real-time image data is processed by feature fusion network to obtain workpiece surface feature data; the workpiece surface feature data is then input into a pre-trained pose estimation model, which uses a particle filter framework to iteratively update the workpiece pose based on the observation likelihood function to generate workpiece pose prediction parameters. The second module is used to construct a multi-objective path planning model based on the workpiece pose prediction parameters. The multi-objective path planning model aims to minimize the gripping path length and the joint motion amplitude. It uses a dynamic programming algorithm to globally optimize the gripping path. The dynamic programming algorithm introduces state transition costs and heuristic evaluation functions to improve path search efficiency. Based on the multi-objective path planning model, it outputs the optimal gripping path data. The third module is used to establish a hierarchical grasping control model based on the optimal grasping path data. The hierarchical grasping control model includes a strategy layer, an adjustment layer, and an execution layer. The strategy layer performs global grasping planning based on the workpiece pose prediction parameters, the adjustment layer performs local path correction based on the optimal grasping path data, and the execution layer implements trajectory tracking of the robotic arm based on a visual servo control algorithm. The hierarchical grasping control model outputs grasping control commands to achieve visual feedback grasping control of the target workpiece.

Images