Robot automatic grabbing path planning method based on visual identification

Abstract

The invention discloses a robot automatic grabbing path planning method based on visual identification, and relates to the technical field of intelligent grabbing. The method comprises the following steps: acquiring multi-view visual data of a target scene, and generating a scene three-dimensional compact reconstruction model through a multi-modal image fusion algorithm; performing target detection and feature extraction on the model, and screening an optimal capture point in combination with a visual attention mechanism; constructing a dynamic environment obstacle probability map, updating an obstacle state through time sequence visual tracking, and quantifying an interference weight; an initial grabbing path is planned based on an improved fast expansion random tree algorithm, path smoothness constraints and robot joint movement limit parameters are introduced, and path nodes are optimized through a Bezier curve; visual servo feedback and path deviation prediction are fused, path parameters are corrected in real time, and a continuous movement track is generated. The method effectively adapts to the dynamic environment, gives consideration to path safety, smoothness and mechanical arm motion characteristics, and remarkably improves the grabbing success rate and operation reliability.

Description

Technical Field

[0001] This invention relates to the field of intelligent grasping technology, and more specifically to a method for automatic grasping path planning for robots based on visual recognition. Background Technology

[0002] In fields such as industrial automation and intelligent logistics, robotic automated grasping technology is a core support for tasks such as material sorting and precision assembly, and its performance directly determines production efficiency and operational accuracy. As application scenarios become increasingly complex, traditional grasping solutions are gradually revealing significant limitations: most solutions rely on single visual data (such as RGB images alone) to construct scene models, making it difficult to accurately reconstruct the three-dimensional shape and spatial position of objects, resulting in large target recognition errors. This is especially true in environments with occlusion or changing lighting, where the grasping success rate drops significantly. Furthermore, existing solutions often select grasping points based on simple geometric features (such as the object's center), without considering key factors such as surface curvature and visual visibility. This can easily lead to problems such as unstable grasping and slippage, failing to meet the high reliability requirements of precision manufacturing and other similar scenarios.

[0003] Obstacle handling in dynamic environments is another core challenge in robot grasping path planning. Traditional path planning algorithms (such as basic A* and artificial potential field methods) often assume static obstacle distribution and can only handle fixed obstacle scenarios. However, in real industrial environments, dynamic obstacles such as materials transported by conveyor belts and moving auxiliary equipment are common. Existing solutions lack the ability to track obstacle movement trends and quantify interference, often resulting in path-obstacle collisions and frequent pauses for adjustments. This not only reduces operational efficiency but also increases the risk of equipment damage. Furthermore, while some solutions introduce dynamic obstacle avoidance mechanisms, they fail to integrate obstacle interference with the robot's own motion constraints (such as joint angle limits and motion smoothness), leading to problems such as joint jamming and discontinuous motion trajectories in the planned path, affecting the lifespan of the robotic arm and grasping accuracy.

[0004] Chinese patent (CN120697040A) discloses a workpiece grasping method based on visual feedback, which constructs a complete system of multi-view perception and hierarchical control. It acquires images through front-view, side-view, and top-view cameras, and extracts workpiece surface features by combining feature fusion networks, effectively making up for the deficiency of incomplete information in single-view images. Pose estimation adopts a particle filter framework and a multi-dimensional observation likelihood function, which integrates contour, texture, feature points, and occlusion information to improve the robustness of pose prediction in dynamic environments. Path planning takes the shortest path and the minimum joint motion as dual objectives, and introduces an adaptive state transition cost and a dynamic heuristic evaluation mechanism to balance path efficiency and robotic arm motion safety. However, this solution still has obvious limitations: multi-view feature fusion only focuses on the workpiece itself and does not fully consider the 3D reconstruction accuracy of the scene environment, resulting in insufficient target separation capability in complex backgrounds; although dynamic obstacle tracking is involved in path planning, the interference weight of obstacles on the path is not quantified, making it difficult to accurately cope with obstacle avoidance requirements in highly dynamic scenarios; deviation correction relies on adaptive adjustment of the local planning window and lacks specific constraints on path smoothness, which can easily lead to jitter or stuttering in the robotic arm's motion trajectory.

[0005] Chinese patent (CN119238538A) discloses a vision-based path planning method for a robotic arm for sorting items. This method addresses image quality and dynamic target processing issues in complex scenarios. It automatically controls the light source through scene complexity analysis and uses an image quality evaluation model to filter target images, effectively mitigating the impact of lighting changes on visual recognition. It employs the YOLOv4 algorithm and RGB-D image fusion to distinguish items, and uses a long short-term memory neural network to predict the positions of items to be sorted and obstacles, improving target localization accuracy in dynamic scenarios. A digital twin model of the robot sorting system is introduced to verify path feasibility, and a genetic algorithm optimizes the robotic arm's movement speed and path, reducing the probability of invalid grasps. However, this solution still has shortcomings. Image fusion is limited to RGB and depth data, limiting its ability to depict fine features such as surface curvature. Path planning does not consider the limits of robot joint movement and path smoothness constraints, focusing optimization on efficiency improvement without comprehensively considering the stability and energy consumption of the robotic arm. Feedback optimization is based only on macroscopic indicators such as grasping speed and accuracy, lacking a real-time closed-loop deviation prediction and correction mechanism, making it difficult to handle sudden minor deviations during grasping.

[0006] Meanwhile, the lack of real-time planning and closed-loop correction capabilities further restricts the practical application of robot grasping technology. Existing technologies often adopt a one-time planning combined with a fixed execution mode, without considering deviations that occur during the grasping process, lacking visual servo feedback and deviation prediction mechanisms, and making it difficult to adjust path parameters in real time. Mainstream path optimization algorithms have problems such as strong sampling randomness and high path redundancy. The planned paths often have redundant inflection points, which not only increases motion time, but also causes end effector posture fluctuations due to path unevenness, affecting grasping stability.

[0007] Therefore, there is an urgent need for a high-precision grasping path planning scheme that integrates multimodal perception, dynamic obstacle adaptation, and real-time closed-loop correction, which has become a key requirement for breaking through the current technical bottleneck. Summary of the Invention

[0008] To address the aforementioned technical problems, this application discloses a visual recognition-based automatic grasping path planning method for robots, specifically including:

[0009] Acquire multi-view visual data of the target scene, including RGB images, depth images and point cloud data, and generate a three-dimensional dense reconstruction model of the scene through a multimodal image fusion algorithm;

[0010] The three-dimensional dense reconstruction model is subjected to target detection and feature extraction to identify the three-dimensional contour, surface curvature distribution and grasping candidate region of the target object, and the optimal grasping point is selected based on the visual attention mechanism.

[0011] Construct a dynamic environmental obstacle probability map, update the obstacle position and movement trend through a time-series visual tracking algorithm, and quantify the interference weight of obstacles on the grasping path;

[0012] The initial grasping path is planned based on the improved fast expanding random tree algorithm. Path smoothness constraints and robot joint motion limit parameters are introduced, and path nodes are optimized by Bézier curves.

[0013] By integrating visual servo feedback and path deviation prediction, path parameters are corrected in real time to generate a continuous motion trajectory of the robot's end effector from the initial posture to the optimal grasping point.

[0014] Preferably, the generation of a three-dimensional dense reconstruction model of the scene using a multimodal image fusion algorithm includes:

[0015] Perform white balance correction and Gaussian filtering noise reduction on RGB images, and extract color feature vectors. Hole filling and outlier removal are performed on the depth image to generate a depth feature matrix. Voxel filtering downsampling is performed on the original point cloud data to preserve geometric feature points; a multimodal fusion criterion function is constructed. ,in The registration loss between the RGB image and the depth image is... The feature matching loss between RGB images and point cloud data, To minimize the geometric consistency loss between depth images and point cloud data, the fusion criterion function is minimized using gradient descent. To optimize the registration accuracy of multimodal data, a three-dimensional dense mesh model of the scene is generated using the Poisson reconstruction algorithm.

[0016] Preferably, the three-dimensional dense reconstruction model performs target detection and feature extraction, identifying the three-dimensional contour and surface curvature distribution of the target object, including:

[0017] A point cloud-based YOLOv8 model is used to perform target detection on a 3D dense reconstruction model, and the bounding box coordinates and category labels of the target objects are output.

[0018] The target object point cloud and the background point cloud are segmented by a region growing algorithm, and the three-dimensional contour line of the target object is extracted. The sampling interval of the contour line is a preset small length.

[0019] Calculate the maximum principal curvature of each vertex on the surface of the target object. and minimum principal curvature Through formula Calculate the curvature difference coefficient to generate a heat map of the curvature distribution on the surface of the target object.

[0020] Preferably, the determination of the grasping candidate region by the three-dimensional contour and surface curvature distribution includes:

[0021] Calculate the minimum bounding cube of the target object based on the 3D contour lines to determine the object's spatial orientation and center coordinates;

[0022] Set curvature smoothness threshold Through formula Regions that meet certain criteria are selected as candidate smooth regions, where the smoothness threshold is... Based on the clamping stability requirements of the end effector, this formula directly eliminates edges and sharp corners with abrupt curvature changes.

[0023] Perform area screening on candidate flat areas and set an area threshold. The area threshold This is a preset multiple of the clamping area of ​​the end effector. The actual area of ​​the candidate flat region is determined by the formula. Filter regions that meet the criteria, and use each region that meets the criteria as a candidate region for capture. Record the center coordinates, normal vector, and extension range of each candidate region.

[0024] Preferably, the step of selecting the optimal grasping point through the visual attention mechanism includes:

[0025] Constructing a visual attention evaluation function ,in This is a comprehensive evaluation score for visual attention. As a curvature smoothing evaluation factor, As the evaluation factor for the central location, As a visual visibility evaluation factor;

[0026] Through the formula: ,in The curvature attenuation coefficient, The curvature difference coefficient. This represents the straight-line distance from the center of the candidate region to the center of the object. The maximum size of the object. The occlusion rate of the candidate region from the camera's perspective;

[0027] Evaluation function based on visual attention The candidate regions are sorted in descending order of function value. A preset number of regions with the highest function value ranking are selected, and the geometric center point of each region is used as the candidate capture point. The three-dimensional coordinates and normal vector of the candidate capture point are output.

[0028] Preferably, the construction of the dynamic environmental obstacle probability map includes:

[0029] The target scene space is divided into a uniform three-dimensional voxel grid, and the voxel size is consistent with the minimum motion step of the robot end effector.

[0030] The occupancy status of each voxel grid is determined based on multi-view visual data. The occupancy status includes free state, occupied state and unknown state.

[0031] Occupied voxels are assigned initial high probability values, free voxels are assigned initial low probability values, and unknown voxels are assigned initial intermediate probability values.

[0032] The occupancy probability of each voxel is updated using Bayesian filtering, as shown in the formula: ,in The voxel occupancy probability of the current frame. This represents the voxel occupancy probability of the previous frame. The conditional probability of observing an obstacle in the occupied state. Given the conditional probability of observing obstacles in a free state, generate a dynamic environment obstacle probability map.

[0033] Preferably, the step of updating the obstacle position and movement trend using a temporal visual tracking algorithm, and quantifying the interference weight of the obstacle on the grasping path, includes:

[0034] The Kalman filter algorithm is used to perform temporal tracking on obstacle objects with an occupancy probability greater than a preset threshold, and an obstacle motion state model is established. The formula is: ,in This is the state vector representing the position and velocity of the obstacle in the current frame. Here is the state transition matrix. This is the obstacle state vector from the previous frame. To control the input matrix, For control vectors, This is the process noise vector;

[0035] Based on observation model Update obstacle status, where This is the visual observation vector for the current frame. For the observation matrix, To observe the noise vector, predict the location distribution of obstacles within a preset number of frames in the future;

[0036] Through formula Calculate the interference weights, where The interference weight of obstacles on the path, The shortest straight-line distance between the obstacle and the planned path. The distance attenuation coefficient is used in the path cost function to quantify the degree of influence of obstacles on path feasibility.

[0037] Preferably, the initial crawling path planning based on the improved fast expanding random tree algorithm includes:

[0038] Let the robot's initial posture be the starting point and the optimal grasping point be the target point. Construct the state space of the robot's joint space. ,in This is the joint space state vector. The number of robot joints. For the first The angle of each joint;

[0039] An improved sampling strategy for the fast expanding random tree algorithm is proposed, which sets a sampling probability bias coefficient near the target point to increase the sampling density in the target area.

[0040] Define path cost function ,in The total cost of the path. , , These are the cost coefficients, The spatial length of the path. This is the sum of the interference weights of all obstacles along the path. Energy consumption for robot joint movement;

[0041] The path tree generated by the greedy algorithm is pruned and optimized, and the path with the lowest cost is retained as the initial crawling path, while the number of path nodes is controlled within a preset range.

[0042] Preferably, the step of introducing path smoothness constraints and robot joint motion limit parameters, and optimizing path nodes using Bézier curves, includes:

[0043] Set the angle limits, angular velocity limits, and angular acceleration limits for each joint of the robot as hard constraints for path optimization;

[0044] Define the path smoothness constraint, as shown in the formula: ,in As an evaluation index for path smoothness, For path in parameters The second derivative at that point, For the parameters of the Bézier curve, minimize To ensure a smooth path;

[0045] A cubic Bézier curve is used to fit the nodes of the initial crawling path. The formula for the Bézier curve is: ,in For parameters The corresponding path point coordinates, The coordinates of the control nodes of the path. To control the number of nodes, These are Bessel basis functions;

[0046] The control points of the Bézier curve are optimized using the gradient descent algorithm to ensure that the optimized path satisfies the joint motion limit constraints and smoothness constraints.

[0047] Preferably, the real-time correction of path parameters by fusing visual servo feedback and path deviation prediction includes:

[0048] The robot uses a vision sensor at its end effector to collect image data in real time during the grasping process and calculates the positional deviation between the end effector and the optimal grasping point. and attitude deviation ,in This is a three-dimensional position deviation vector. This is a three-dimensional attitude deviation vector;

[0049] A path deviation prediction model is constructed, which uses a long short-term memory network to learn historical deviation sequences and outputs the predicted deviation value for the future preset control period. and ,in This is the position deviation prediction vector. This is the attitude deviation prediction vector;

[0050] Through formula Define the correction amount, where This is the joint angle correction vector. The pseudo-inverse of the Jacobian matrix for the robot. For predicting weighting coefficients;

[0051] Correction amount The joint angle parameters are superimposed on the original path, and the path parameters are updated at preset time intervals to achieve real-time closed-loop correction of the grasping process.

[0052] Compared with the prior art, the technical solution of this application has the following technical effects:

[0053] This invention constructs a three-dimensional dense reconstruction model by fusing RGB images, depth images, and point cloud data, effectively compensating for the information deficiencies of single visual data and accurately restoring the three-dimensional contour and spatial posture of the target object. By combining surface curvature analysis and visual attention mechanisms to select the optimal grasping point and avoiding unstable areas such as edges and occlusions, the rationality and reliability of grasping point selection are greatly improved, ensuring the accuracy of target recognition and grasping preparation in complex environments.

[0054] This invention constructs a dynamic obstacle probability map and updates the obstacle motion state through a time-series tracking algorithm, quantifies the interference weight, and enables path planning to respond to moving obstacles in real time. It deeply integrates obstacle interference with path planning, avoids the collision risk caused by static obstacle avoidance in traditional solutions, ensures the safety and continuity of robot operation in dynamic and complex scenarios, and improves adaptability to actual industrial environments.

[0055] This invention is based on an improved fast expanding random tree algorithm, which combines path smoothness constraints and robot joint motion limit parameters. By optimizing path nodes through Bézier curves, it effectively reduces path redundancy inflection points and joint jamming. The path cost function comprehensively considers length, obstacle interference, and energy consumption, and the planned path is both efficient and short, and conforms to the motion characteristics of the robotic arm, thus extending the service life of the equipment.

[0056] This invention integrates visual servo feedback and LSTM deviation prediction model to capture and predict position and posture deviations in real time and dynamically adjust path parameters through a closed-loop correction mechanism. It effectively offsets the effects of factors such as robotic arm motion errors and slight object displacement, ensuring continuous and stable motion of the end effector from the initial posture to the grasping point, and significantly improving the consistency and success rate of grasping actions.

[0057] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the preferred embodiments of this application are described in detail below with reference to the accompanying drawings.

[0058] The above and other objects, advantages and features of this application will become more apparent to those skilled in the art from the following detailed description of specific embodiments in conjunction with the accompanying drawings. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In all drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0060] Based on the description of the figures and their corresponding technical content in the document, the titles of the figures are as follows:

[0061] Figure 1 This is a vision-based method for automatic robot grasping path planning.

[0062] Figure 2 Side and back views of a 3D dense reconstruction model of the target object (teddy bear);

[0063] Figure 3 A multi-view labeled map for identifying candidate regions for a target object (teddy bear);

[0064] Figure 4 This is a flowchart illustrating the process architecture for obtaining the optimal grasping point based on visual recognition.

[0065] Figure 5 A multi-view annotated image after selecting the optimal gripping point for the target object (teddy bear);

[0066] Figure 6 A flowchart illustrating the improved RRT algorithm for initial crawling path planning;

[0067] Figure 7 A schematic diagram of the probability map and trajectory annotation of obstacles in a dynamic environment;

[0068] Figure 8 A flowchart of the process architecture generated for the optimal crawling path;

[0069] Figure 9 This is a schematic diagram showing the node distribution and obstacle locations along the initial grabbing path;

[0070] Figure 10 This is a diagram comparing the obstacle avoidance performance of the original path and the optimized path. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. In the following description, specific details such as specific configurations and components are provided merely to help fully understand the embodiments of this application. Therefore, those skilled in the art should understand that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. In addition, for clarity and brevity, descriptions of known functions and structures are omitted in the embodiments.

[0072] It should be understood that the phrase "an embodiment" or "this embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "an embodiment" or "this embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0073] Furthermore, reference numerals and / or letters may be repeated in different examples within this application. Such repetition is for the purpose of simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or settings discussed.

[0074] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist simultaneously. The term " / and" in this article describes another type of relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " in this article generally indicates that the related objects before and after it are in an "or" relationship.

[0075] In this article, the term "at least one" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, "at least one of A and B" can mean: A exists alone, A and B exist simultaneously, or B exists alone.

[0076] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion.

[0077] Example 1

[0078] This embodiment mainly describes a visual recognition-based automatic grasping path planning method for robots, such as... Figure 1 As shown, it specifically includes:

[0079] Acquire multi-view visual data of the target scene, including RGB images, depth images and point cloud data, and generate a three-dimensional dense reconstruction model of the scene through a multimodal image fusion algorithm;

[0080] The three-dimensional dense reconstruction model is subjected to target detection and feature extraction to identify the three-dimensional contour, surface curvature distribution and grasping candidate region of the target object, and the optimal grasping point is selected based on the visual attention mechanism.

[0081] Construct a dynamic environmental obstacle probability map, update the obstacle position and movement trend through a time-series visual tracking algorithm, and quantify the interference weight of obstacles on the grasping path;

[0082] The initial grasping path is planned based on the improved fast expanding random tree algorithm. Path smoothness constraints and robot joint motion limit parameters are introduced, and path nodes are optimized by Bézier curves.

[0083] By integrating visual servo feedback and path deviation prediction, path parameters are corrected in real time to generate a continuous motion trajectory of the robot's end effector from the initial posture to the optimal grasping point.

[0084] Furthermore, the visual recognition of this application is described in detail. Through multi-source data fusion and feature extraction, the target object is accurately located and the optimal grasping point is selected, specifically as follows:

[0085] The robot is equipped with a multi-view vision acquisition system that simultaneously collects three core types of visual data from the target scene: RGB images, depth images, and point cloud data. The RGB images are acquired using a 12-megapixel industrial camera with a resolution of 4032×3024, accurately capturing the color and texture details of the target object and providing fundamental data for appearance dimension matching. The depth images are acquired using a TOF depth camera with a measurement range of 0.3-10m and a depth accuracy of ±1%, directly outputting the distance information from each point on the object's surface to the camera, compensating for the lack of depth dimension in two-dimensional images. The point cloud data is acquired by a 3D laser scanner with a point cloud density of 1000 points / square centimeter, recording the spatial positions of a massive number of points on the object's surface in three-dimensional coordinates (X,Y,Z), directly reflecting the object's geometric shape.

[0086] After the three types of raw data are collected, targeted preprocessing operations are required to eliminate noise, fill defects, and simplify the data, laying the foundation for subsequent fusion modeling. For RGB images, white balance correction is first performed, using the grayscale world algorithm to adjust the gain of the R, G, and B channels of the image to eliminate color temperature deviations under different lighting conditions and ensure that the inherent color characteristics of objects are not affected by lighting. Then, a 5×5 Gaussian filter kernel is used for noise reduction, with the Gaussian kernel function being... By weighted averaging of the neighborhood around each pixel, high-frequency noise (such as ambient light and sensor noise) in the image is effectively filtered out, while key information such as the object's edge contour is preserved to the greatest extent. A 128-dimensional color feature vector F_c is extracted through principal component analysis (PCA). The vector covers the core distribution features of the object in the RGB color space, providing a color dimension constraint for multimodal data registration.

[0087] Due to the limitations of the measurement principle of TOF cameras, depth images are prone to containing void regions (such as areas obscured by objects) and outliers (such as errors from long-distance measurements), requiring repair and purification. A region-growing-based void-filling algorithm is employed, using the effective depth value at the edge of the void region as the seed point. By judging the depth continuity of neighboring pixels (with a depth difference threshold set to 0.5mm), void regions are gradually filled to ensure the integrity of the depth image. Simultaneously, a statistical filtering algorithm is used to remove outliers, setting the neighborhood window size to 11×11 and calculating the mean depth value within the window. and standard deviation It will exceed Pixels within a certain range are identified as outliers and removed, generating a 64×64 resolution depth feature matrix. Each element in the matrix corresponds to the corrected depth value of the original image pixel, providing accurate distance data support for three-dimensional spatial location calculation;

[0088] Raw point cloud data typically contains millions or even tens of millions of points, including a large amount of redundant information. Direct processing would lead to excessive computational load and low efficiency. A voxel filtering downsampling algorithm is used to simplify the data. The voxel grid size is set to 1mm × 1mm × 1mm. The geometric center of all point cloud data within each voxel grid is calculated, and only this center is retained as a representative point. While ensuring that key geometric features of the object (such as edges and surfaces) are not lost, the amount of point cloud data is reduced to about 1 / 10 of the original, significantly improving the real-time performance of subsequent data processing. A normal vector estimation algorithm is used to calculate the normal vector of each point cloud, providing a foundation for subsequent 3D contour extraction and curvature analysis.

[0089] After preprocessing the three types of data, a multimodal fusion criterion function is constructed to achieve accurate registration and fusion of RGB images, depth images, and point cloud data; the fusion criterion function is defined as follows: ,in: The registration loss between the RGB image and the depth image is calculated by measuring the SIFT feature point matching error between the two types of images. The matching error is measured using Euclidean distance, and the formula is as follows: , To match the number of feature points, These are the coordinates of feature points in an RGB image. (Coordinates of the feature points corresponding to the depth image); The feature matching loss between RGB image and point cloud data is calculated by projecting the point cloud data onto the RGB image plane and calculating the cosine similarity between the projected points and the color feature vectors of the RGB image. The complement of the mean similarity is the matching loss. The geometric consistency loss between the depth image and the point cloud data is calculated by summing the squares of the differences between the depth values ​​in the depth image and the corresponding Z-coordinates in the point cloud data. The formula is as follows: , The number of effective projection points, For depth image depth values, For point cloud data coordinate;

[0090] Gradient descent method is used to analyze the fusion criterion function. Iterative optimization was performed with a learning rate of 0.001 and 1000 iterations. By continuously adjusting the spatial transformation parameters (translation and rotation matrices) of the three types of data, the F-value converged to its minimum (convergence threshold set to 0.01), at which point the three types of data achieved optimal registration in spatial location. Based on this, the Poisson reconstruction algorithm was used for 3D dense reconstruction of the scene. This algorithm treats the registered multimodal data as a scalar field in 3D space, constructs a continuous 3D surface model, and generates a 3D dense reconstruction model of the scene with a mesh resolution of 0.5mm.

[0091] like Figure 1 The image shows a visualization of the results of the above fusion and reconstruction process, rendered using a pseudo-color rendering method. Figure 1In the model, taking a blue teddy bear as an example, the surface transitions, edge contours, and uneven structures are accurately reproduced, with no obvious holes or distortion areas. The surrounding environment is displayed in gray, allowing for a direct distinction between the target object and the background through color contrast. Simultaneously, the model overlays key dimensions (e.g., length × width × height: 150mm × 80mm × 60mm), center point coordinates (X: 200mm, Y: 150mm, Z: 100mm), and surface normal vector distribution (marked with red arrows), comprehensively reflecting the three-dimensional feature information of the target object. The average registration error of this densely reconstructed 3D model is less than 0.3mm, and its geometric accuracy meets the requirements for precise grasping.

[0092] based on Figure 1 A 3D dense reconstruction model is used to further accurately locate target objects and analyze their surface geometric characteristics. A point cloud-based YOLOv8 model is employed for target detection. This model introduces a point cloud feature fusion module on top of the original YOLOv8, significantly improving target recognition accuracy in complex scenes by combining the geometric features of the point cloud (such as normal vectors and curvature) with the color features of the RGB image. The model's input consists of registered point cloud data and RGB feature vectors. Multi-scale features are extracted through the backbone network (C2f module), and feature fusion is performed using the neck network (SPPF module). The detection head outputs the bounding box coordinates, class label, and confidence score of the target object. A confidence threshold of 0.85 is set. When the confidence score of the detection result is higher than this threshold, it is considered a valid target object, and its 3D bounding box coordinates (e.g., X: 125-275mm, Y: 110-190mm, Z: 70-130mm) and class label are output.

[0093] After target object detection, the target object point cloud and background point cloud are segmented using a region growing algorithm. The seed point for region growing is set as the point cloud within the target object's bounding box. The growth criteria are: the angle between the normal vector of the neighboring point and the seed point is less than 10°, and the spatial distance is less than 2mm. Through iterative growing, all point clouds meeting these conditions are classified as target object point clouds, while the remaining point clouds are identified as background point clouds, achieving accurate separation between the target object and the background. Subsequently, 3D contour extraction is performed based on the segmented target object point cloud. The α-shape algorithm (α=2mm) is used to extract the edge contour lines of the triangular mesh by constructing a Delaunay triangulation of the point cloud. The contour lines are discretized at a sampling interval of 0.5mm, resulting in a continuous 3D contour composed of over 1000 sampling points. This contour accurately reflects the outer boundary shape of the target object, providing a basis for subsequent minimum bounding box calculation and grasping region selection.

[0094] To analyze the gripping adaptability of a target object's surface, its surface curvature distribution needs to be calculated. For each vertex of the target object's point cloud, this is done by... Nearest neighbor points ( =20) Construct the local covariance matrix (in (where the geometric center is the nearest neighbor), perform eigenvalue decomposition on the covariance matrix to obtain three eigenvalues. The corresponding principal curvature ; through formula Calculate the curvature difference coefficient for each vertex. The coefficient ranges from [0,1]. The closer the value is to 0, the smoother the surface is. The closer the value is to 1, the more drastic the surface curvature changes (such as edges and corners).

[0095] exist Figure 2 In the image, the surface curvature distribution of the target object is visually presented in the form of a heat map, where the curvature difference coefficient is... (Area with a gentle surface) (Surface with moderate curvature) >0.6 (Abrupt changes in surface curvature, such as corners and edges); As can be clearly seen from the figure, the center of the front, top, and lower side of the target object are flat areas, while the four corners and edges of the object are protruding areas. This distribution feature provides a clear geometric basis for the subsequent selection of candidate areas for grasping. Flat areas are more suitable for end effector clamping and can effectively avoid problems such as slippage and unstable clamping during the grasping process.

[0096] based on Figure 2 The extracted feature information, such as the 3D contour and surface curvature distribution of the target object, is used to further filter candidate grasping regions and determine the optimal grasping point. Figure 2 The minimum bounding box of the target object is calculated from the 3D contour data. This process uses Principal Component Analysis (PCA) to determine the direction of the object's principal coordinate axes, and then calculates the maximum and minimum values ​​of the object's point cloud in the X, Y, and Z dimensions along these axes, thus obtaining the vertex coordinates of the minimum bounding box. The calculated coordinates of the eight vertices of the minimum bounding box are as follows: (125,110,70) (275,110,70) (275,190,70) (125,190,70) (125, 110, 130) (275,110,130) (275,190,130) (125,190,130) The circumscribed cuboid can be used to determine the spatial orientation (angle between the principal coordinate axis and the world coordinate system) and the center coordinates O (200,150,100) of the target object, providing a spatial reference for the position selection of the grasping area.

[0097] Set curvature smoothness threshold This threshold is determined based on the clamping stability requirements of the end effector. The end effector used is a two-finger parallel gripper with a clamping surface size of 30mm × 20mm and a clamping material of rubber. The threshold value was determined through mechanical simulation analysis, when the surface curvature difference coefficient C... At a value of 0.3, the contact area between the gripper and the object surface is ≥80%, the gripping force is evenly distributed, and slippage can be effectively prevented. Therefore, it is set... =0.3. According to formula C ,from Figure 2 The target object surface is filtered to identify all smooth regions that meet the criteria; these regions are... Figure 3 The area was marked with a light semi-transparent region, resulting in four initial flat areas, located in the center of the front, the center of the top, the center of the back, and the lower part of the side of the object.

[0098] Because the end effector's gripping surface has a minimum effective gripping area requirement (if the gripping area is too small, it will lead to insufficient gripping force and cause gripping failure), it is necessary to screen the initial flat area by area. An area threshold is set. , The clamping area is 1.2 times that of the end effector (with a certain margin to ensure clamping stability). Therefore, the clamping area of ​​the end effector is 30mm × 20mm = 600mm². =600 × 1.2 = 720 mm². The surface area of ​​each initially flat region was calculated (using the grid integration method, dividing the region into several small triangles, calculating the area of ​​each triangle and summing the results), and the surface area was selected. The region is used as a candidate region for crawling.

[0099] Calculations show that the surface area of ​​the central area on the front is 850 mm², the central area on the top is 780 mm², the central area on the back is 820 mm², and the lower area on the side is 650 mm² (less than). (Those that were rejected) were selected as three candidate regions for grabbing, denoted as candidate region 1 (center of the front), candidate region 2 (center of the top), and candidate region 3 (center of the back). Figure 3 The text is marked with a red dashed box. Figure 3In the diagram, each dashed box is labeled with the key parameters of the corresponding candidate region: Candidate region 1 has the following parameters: center coordinates A (200, 130, 100), normal vector n1 (0, 1, 0) (pointing outwards), extension range (X: 150-250 mm, Y: 110-150 mm, Z: 80-120 mm), and surface area 850 mm²; Candidate region 2 has the following parameters: center coordinates B (200, 150, 120), normal vector n2 (0, 0, 1), extension range (X: 160-240 mm, Y: 120-180 mm, Z: 110-130 mm), and surface area 780 mm²; Candidate region 3 has the following parameters: center coordinates C (200, 170, 100), normal vector n3 (0, -1, 0), extension range (X: 150-250 mm, Y: 150-190 mm, Z: 80-120 mm), and surface area 820 mm².

[0100] To select the optimal grab point from three candidate grab regions, a visual attention evaluation function is constructed, comprehensively considering three key factors: the curvature smoothness of the region, the rationality of the center position, and visual visibility. The evaluation function is defined as follows: Where R is the comprehensive evaluation value of visual attention (the value ranges from [0, ..., ...). ]), As a curvature smoothing evaluation factor, As the evaluation factor for the central location, It is a factor for evaluating visual visibility.

[0101] The calculation methods for each evaluation factor are as follows: Curvature smoothness evaluation factor The calculation is performed using an exponential decay function, and the formula is as follows: ,in The curvature attenuation coefficient, This represents the average curvature difference coefficient of the candidate regions. Through calculation, the average curvature difference coefficient of candidate region 1... =0.15, then The average of candidate region 2 =0.20, then The average of candidate region 3 =0.18, then . The larger the value, the smoother the curvature of the region and the better the clamping stability.

[0102] Central location evaluation factor Similarly, the exponential decay function is used for calculation, and the formula is as follows: ,in From the center of the candidate region to the center of the object The straight-line distance The maximum size of the object (i.e., the length of the spatial diagonal of the smallest circumscribed cuboid); calculate the distance from the center of each candidate region to... Distance:

[0103]

[0104]

[0105]

[0106] Maximum size of object ,therefore, ; The larger the value, the closer the candidate area is to the center of the object, and the more uniform the force on the object is during the grasping process, which can reduce the problem of unstable clamping caused by the shift of the center of gravity.

[0107] Visual visibility evaluation factor The formula is ,in Let represent the occlusion rate of the candidate regions from the camera's perspective. Assuming the robot's vision acquisition device is mounted above the end effector, the camera's field of view is 60°, and the shooting distance is 300mm, the visibility of each candidate region is calculated using a ray tracing algorithm: Candidate region 1 (center of the front) has no occlusion, o1=0, then... =1.0; Candidate region 2 (center of the top surface) is unobstructed, o2=0, then =1.0; Candidate region 3 (middle of the back) is occluded by the object's own structural part (tail), with an occlusion rate o3 = 0.3, then =0.7; The larger the value, the easier it is for the candidate region to be monitored by the visual sensor during the grasping process.

[0108] Calculate the R-value for each candidate region based on the visual attention evaluation function:

[0109] Candidate Region 1: Candidate Region 2: Candidate region 3: ;

[0110] Candidate region 1 is sorted in descending order of R value. Ranked first, candidate region 2 Ranked second, candidate region 3 Ranked third. The candidate region 1, ranked first, is selected as the optimal grab region. Its geometric center point is the optimal grab point, and its 3D coordinates coincide with the center coordinates of candidate region 1. The normal vector is .

[0111] like Figure 4As shown, the above filtering results are visualized, with the optimal capture point marked by a solid yellow dot. ,like Figure 5 As shown, the three-dimensional coordinates, normal vector, and visual attention comprehensive evaluation value R=1.532 of the point are clearly labeled next to the dot. Meanwhile, the figure retains... Figure 3 The three candidate gripping regions are marked with different colors to distinguish the R-values ​​(candidate region 1 is the darkest, and candidate region 3 is the lightest), intuitively demonstrating the selection logic and advantages of the optimal gripping point. This optimal gripping point has three major characteristics: smooth curvature, centered position, and high visual visibility. It can maximize the gripping stability of the end effector and the monitorability of the gripping process, providing accurate target endpoint data for subsequent path planning.

[0112] After determining the optimal grasping point, it is necessary to dynamically perceive and track the movement trends of obstacles in the target scene, construct a dynamic environmental obstacle probability map, quantify the interference weight of obstacles on the grasping path, and the relevant results are processed through... Figure 5 Visualization provides precise obstacle constraints for path planning.

[0113] The target scene space is divided into a uniform three-dimensional voxel grid according to the minimum movement step length of the robot's end effector (set to 1mm). The range of the voxel grid covers the robot's movement space (X: 0-500mm, Y: 0-500mm, Z: 0-300mm). Therefore, the entire scene is divided into 500×500×300=75,000,000 voxels, and the size of each voxel is 1mm×1mm×1mm, ensuring that the positional relationship between the end effector's movement space and obstacles can be accurately represented.

[0114] The occupancy state of each voxel mesh is determined based on multi-view visual data (preprocessed RGB images, depth images, and point cloud data). The occupancy state is divided into three categories: free state, occupied state, and unknown state.

[0115] Free state: When there is no point cloud data in the voxel mesh and the depth value of the corresponding position in the depth image is greater than the robot's maximum range of motion (300mm), it is determined to be in a free state, indicating that there are no obstacles in the area and the end effector can pass freely;

[0116] Occupied state: When the voxel mesh contains point cloud data (non-target object point cloud), or the depth value of the corresponding position in the depth image is less than a preset threshold (5mm), it is determined to be occupied state, indicating that there is an obstacle in the area;

[0117] Unknown state: When there is no point cloud data in the voxel grid, but the depth value of the corresponding position in the depth image is between 5-300mm, it is determined to be an unknown state, indicating that the existence of obstacles in the area cannot be determined by the current visual data.

[0118] Assign initial probability values ​​to each occupied state: assign initial probability to free voxels. =0.2 (low probability of obstacle), the occupied state voxel is assigned an initial probability. =0.8 (high probability of obstacles), unknown state voxels are assigned initial probabilities =0.5 (the probability of the existence of an obstacle is uncertain).

[0119] A Bayesian filtering algorithm is used to update the occupancy probability of each voxel in real time to adapt to changes in obstacle position in a dynamic environment. The core formula of Bayesian filtering is: in: The occupancy probability of the voxel in the current frame (frame t); The occupancy probability of the voxel in the previous frame (frame t-1); The conditional probability of observing an obstacle in the occupied state (set to 0.95, meaning that when the voxel is actually in the occupied state, the probability of the visual sensor observing an obstacle is 95%). The conditional probability of observing an obstacle in a free state (set to 0.05, meaning that when the voxel is actually in a free state, the probability of the visual sensor mistakenly observing an obstacle is 5%).

[0120] A dynamic obstacle probability map is generated by iteratively updating 100 consecutive frames of visual data using Bayesian filtering. Figure 7 This is the visualization result of the map; Figure 7 The code uses grayscale values ​​to represent the occupancy probability of a voxel; the darker the grayscale value, the higher the occupancy probability (black indicates...). =1.0, white indicates =0.0). It is clearly visible in the image that there is a dynamic obstacle (such as a moving conveyor belt or other workpiece) to the left of the target object. Its corresponding voxel region (X: 50-120mm, Y: 100-160mm, Z: 80-120mm) is dark gray, indicating an occupancy probability. A value ≥0.9 indicates an extremely high probability of an obstacle being present in the area; other areas surrounding the target object are mostly white or light gray, indicating a low probability of occupancy. A value of ≤0.3 indicates that these areas are free of obstacles or have an extremely low probability of having obstacles, allowing the end effector to pass safely.

[0121] To accurately track the movement trend of obstacles, the occupancy probability is... ≥0.7 obstacle elements (i.e. Figure 7 The Kalman filter algorithm is used for time-series tracking in the medium-dark gray area. A motion state model and observation model of the obstacle are established to achieve real-time estimation of the obstacle's position and velocity, and prediction of its future position.

[0122] The motion state model of an obstacle is defined as follows: ,in: The state vector of the obstacle in the current frame, including its position. and speed ; Here is the state transition matrix. , The frame interval time is set to 0.01s. For the control input matrix, it is assumed here that there is no external control input to the obstacle. =0; For control vectors, =0; This is the process noise vector, with a mean of 0, used to characterize the uncertainty of the model;

[0123] The observation model is defined as follows: ,in: H is the observation vector of the visual sensor in the current frame, i.e., the observed position of the obstacle; H is the observation matrix. ; The observation noise vector follows a Gaussian distribution with a mean of 0 and is used to characterize the uncertainty of the observation.

[0124] The obstacle observation data of 50 consecutive frames is processed through the prediction-update iterative process of Kalman filtering to update the obstacle state vector in real time. Based on the updated state vector, the obstacle location distribution for the next 5 frames (0.05s) is predicted. Figure 7 In the diagram, the obstacle's existing trajectory is marked with a solid line, while the predicted future trajectory is marked with a dashed line. The trajectory shows that the obstacle is moving at a constant speed along the positive X-axis, with a speed of approximately 10 mm / s. It will continue to move towards the target object in the future, which will interfere with the grasping path and needs to be taken into consideration in path planning.

[0125] To quantify the degree of interference of obstacles on the grasping path, obstacle interference weights are defined. Through formula Calculation, where The shortest straight-line distance between the obstacle and the planned path. This is the distance attenuation factor (set to 50mm). The value range is (0,1]. The smaller, The closer d is to 1, the stronger the interference of obstacles on the path; the larger d is, the stronger the interference. The closer it is to 0, the weaker the interference.

[0126] With optimal capture point Based on the initial robot pose points (200, 130, 100) and (350, 250, 150) (mm), an initial path (represented by a dashed line) is preset. The shortest straight-line distance d = 30mm between this path and the obstacle is calculated. Then, the obstacle's interference weight on this path is determined. This interference weight will serve as an important component of the path cost function, quantifying the impact of obstacles on path feasibility during path planning and ensuring that the planned path can effectively avoid obstacles.

[0127] Based on the optimal grasping point and a dynamic environmental obstacle probability map, an improved Rapid Expanding Random Tree (RRT) algorithm is used to plan the initial grasping path, and Bézier curves are used for path optimization. Specifically:

[0128] Define the starting and target points for path planning. The starting point is the joint space state corresponding to the initial posture of the robot's end effector. Assume the robot is a 6-DOF robotic arm, and the initial joint angle vector is... The corresponding end effector position coordinates are S(350,250,150) (unit: mm); the target point is the optimal grasping point. (200, 130, 100) (unit: mm), the corresponding target joint angle vector is obtained by inverse kinematics.

[0129] State space for constructing robot joint space ,in (i=1,2,...,6) represents the angle of the i-th joint. The angle range of each joint is [-120°,120°]. The angular velocity limit is 30° / s, and the angular acceleration limit is 10° / s². These parameters will serve as constraints for path planning.

[0130] An improvement to the sampling strategy of the traditional Rapid Expanding Random Tree (RRT) algorithm is made by introducing a target region sampling bias coefficient to increase the sampling density near the target point and accelerate path convergence. A sampling probability bias coefficient is set. =0.7, meaning 70% of the sampling points are at the target point. Sampling points are randomly generated within a 50mm radius, with 30% of the sampling points randomly generated throughout the entire joint space. This improved sampling strategy allows the random tree to grow towards the target region more quickly, reducing invalid sampling and path search time, and improving the real-time performance of path planning.

[0131] Define path cost function Taking into account three major factors—path length, total obstacle interference weights, and robot joint motion energy consumption—the formula is as follows: ,in: , , The cost coefficients for path length, total obstacle interference weights, and joint motion energy consumption are determined using the Analytic Hierarchy Process (AHP). =0.4, =0.3, =0.3, ensuring that the weights of each factor on the path cost are reasonable; L is the spatial length of the path, which is the sum of the Euclidean distances between all adjacent nodes on the path; This represents the sum of the interference weights of all obstacles along the path. If the path passes through multiple obstacle regions, the interference weight of each obstacle needs to be calculated and summed. E represents the energy consumption of the robot's joint movements, expressed by the formula... Calculation, where For the first The driving torque of each joint The joint angular velocity, and These represent the start and end times of the path, respectively.

[0132] Based on the improved RRT algorithm and path cost function, initial crawling path planning is performed, such as... Figure 6 As shown:

[0133] Initialize the random tree, setting the starting point... Corresponding joint space state As the root node of the random tree;

[0134] Random sampling points are generated according to the improved sampling strategy. ;

[0135] Search for and in a random tree nearest node ;

[0136] from Towards New nodes growing in the direction The growth step length is set to a joint angle of 5° (corresponding to a spatial displacement of approximately 5mm for the end effector).

[0137] examine Is the corresponding end effector position in the free state region of the obstacle probability map? ≤0.3), if in the free state region, then Add to a random tree;

[0138] Repeat the steps until the random tree grows to the target point. Corresponding joint space state , forming a line from arrive The initial path;

[0139] A greedy algorithm is used to prune and optimize the initial path tree. It traverses all nodes on the path, deletes redundant nodes (such as three consecutive nodes that are approximately collinear and have no obstacles), and retains the path with the minimum cost as the initial crawling path.

[0140] like Figure 7 The diagram shows the planned initial crawling path, marked with solid lines. The path starts from the starting point S(350,250,150), passes through nodes N1(300,220,140), N2(250,180,120), and N3(220,150,110), and reaches the target point. (200, 130, 100), total path length L = 285 mm, total obstacle interference weights =0.698, joint motion energy consumption E=15.6J, total path cost The initial path avoided Figure 7 The system eliminates obstacles in the path and limits the number of path nodes to no more than 20, ensuring both path simplicity and real-time performance.

[0141] To further improve the smoothness of the path and the stability of the robot's motion, path smoothness constraints and robot joint motion limit parameters are introduced. The nodes of the initial grasping path are optimized using Bézier curves, such as... Figure 8 As shown.

[0142] Define the constraints for path optimization:

[0143] Joint movement limit constraints, angle range of each joint ∈[-120°, 120°], angular velocity ≤30° / s, angular acceleration ≤10° / s², these are hard constraints, and the optimized path must strictly adhere to them to avoid joint over-limits that could lead to mechanical damage;

[0144] Path smoothness constraints, defining path smoothness evaluation metrics ,in Let be the second derivative of the path at the parameter τ (τ∈[0,1]). The smaller the value, the more uniform the curvature change of the path, and the higher the smoothness. By minimizing... This ensures that the optimized path has no sharp bends or abrupt changes, reducing impact and vibration during robot movement.

[0145] A cubic Bézier curve is used to fit the nodes of the initial crawling path. The formula for the cubic Bézier curve is: ,in For parameters The corresponding path point coordinates, P0, P1, P2, P3 are the coordinates of the control nodes of the path (P0 is the starting point S, P3 is the target point). (P1 and P2 are intermediate control nodes to be optimized). Let be a cubic Bessel basis function, defined as ,in It is the number of combinations.

[0146] With the goal of minimizing the path smoothness evaluation index Sigma, and combined with joint motion limit constraints, an optimization objective function is constructed as follows: The gradient descent algorithm is used to optimize the intermediate control nodes P1 and P2 of the Bézier curve. The learning rate is set to 0.01, and the number of iterations is 500. By continuously adjusting the coordinates of P1 and P2, the objective function is optimized. It converges to the minimum value while satisfying all constraints.

[0147] like Figure 10 As shown, the optimized crawling path is illustrated. The optimized path is marked with a solid red line in the figure. Figure 9 The optimized path is compared with the initial path. The optimized path is a smooth curve with no obvious inflection points, and the total path length L = 290mm (slightly increased compared to the initial path, but still within a reasonable range). The path smoothness evaluation index... =0.85 (initial path) =3.26, smoothness improved by 74%. Through simulation analysis of joint angles, angular velocities, and angular accelerations, the motion parameters of all joints in the optimized path did not exceed the limit constraints, with a maximum angular velocity of 25° / s and a maximum angular acceleration of 8° / s², meeting the robot's motion performance requirements.

[0148] To address path deviations during the grasping process (such as robot motion errors, slight object displacement, and obstacle position changes), visual servo feedback and path deviation prediction are integrated to perform real-time corrections on the optimized path, ensuring the end effector accurately reaches the optimal grasping point. The correction process and results are verified through... Figure 10 To provide a visual presentation.

[0149] The robot's end effector uses a vision sensor (from the same multi-view vision system used for initial data acquisition) to capture image data in real time during the grasping process (100fps). An image registration algorithm is then used to calculate the optimal grasping point between the end effector and the target grasping point. Positional deviation and attitude deviation Among them, the location is biased. The current position of the end effector and The three-dimensional coordinate difference is calculated using a template matching algorithm, with a positional deviation accuracy of ±0.1mm; attitude deviation The current pose and target pose (normal vector) of the end effector The Euler angle difference of the attitude corresponding to (0,1,0) is obtained by feature point matching and attitude calculation, and the attitude deviation accuracy can reach ±0.1°.

[0150] A path deviation prediction model based on a Long Short-Term Memory (LSTM) network is constructed to learn the changing patterns of historical deviation sequences and predict the deviation value for a preset control period (set to 0.01s). The input to the LSTM model is the deviation sequence of the past 20 frames (…). The output is the predicted deviation value for the next frame. and The model was trained using 1,000 sets of bias data collected in the early stage. After training, the prediction error was ≤5%, and it was able to accurately capture the changing trend of the bias (such as linear growth and periodic fluctuations).

[0151] Define joint angle correction vector Taking into account both the current deviation and the prediction deviation, the formula is: ,in: It is the pseudo-inverse of the robot's Jacobian matrix, obtained by solving the robot's forward kinematics model, and is used to map the position and orientation deviation of the end effector to the joint angle deviation. To predict the weighting coefficients, the influence of the current bias and the prediction bias must be balanced; and These are the combined value of position deviation and the combined value of attitude deviation, respectively. They take into account the current actual deviation and predict future deviation changes in advance by predicting the deviation, thus achieving forward-looking correction.

[0152] Joint angle correction vector The joint angle parameters are superimposed onto the original optimized path, and the path parameters are updated at a preset time interval of 0.01s to form a real-time corrected path. Figure 10 The image shows the actual motion trajectory after real-time correction. The trajectory reveals that during the grasping process, a slight shift in the obstacle's position (actual shift of 2mm) caused a deviation in the initial optimized path. After capturing this deviation through visual servoing feedback, the deviation is combined with the deviation predicted by the LSTM model. The joint angle correction vector is calculated. It also updates path parameters in real time, ensuring that the actual motion trajectory of the end effector accurately matches the optimal gripping point with a deviation controlled within ±0.2mm, thus ensuring the accuracy and stability of the gripping action.

[0153] This embodiment achieves accurate target object identification and optimal grasping point selection through multi-view visual data fusion and 3D reconstruction technology. Combined with dynamic obstacle probability maps and temporal tracking, it effectively avoids environmental interference. Based on an improved algorithm, initial path planning and Bézier curve optimization balance path smoothness and robot joint motion constraints. Real-time deviation correction via visual servo feedback significantly improves the safety and adaptability of the grasping path. This embodiment can achieve automated grasping without human intervention, greatly enhancing the stability and reliability of robot operations in complex dynamic scenarios. It adapts to various grasping task requirements and possesses strong practical value and promising prospects for widespread application.

[0154] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. For those skilled in the art, the present invention can have various modifications and variations. Any changes, modifications, substitutions, integrations, and parameter changes made to these embodiments within the spirit and principles of the present invention, without departing from the principles and spirit of the present invention, through conventional substitutions or to achieve the same function, fall within the scope of protection of the present invention.

Claims

1. A method for automatic robot grasping path planning based on visual recognition, characterized in that, include: Acquire multi-view visual data of the target scene, including RGB images, depth images and point cloud data, and generate a three-dimensional dense reconstruction model of the scene through a multimodal image fusion algorithm; The three-dimensional dense reconstruction model is subjected to target detection and feature extraction to identify the three-dimensional contour, surface curvature distribution and grasping candidate region of the target object, and the optimal grasping point is selected based on the visual attention mechanism. Construct a dynamic environmental obstacle probability map, update the obstacle position and movement trend through a time-series visual tracking algorithm, and quantify the interference weight of obstacles on the grasping path; The initial grasping path is planned based on the improved fast expanding random tree algorithm. Path smoothness constraints and robot joint motion limit parameters are introduced, and path nodes are optimized by Bézier curves. By integrating visual servo feedback and path deviation prediction, path parameters are corrected in real time to generate a continuous motion trajectory of the robot's end effector from the initial posture to the optimal grasping point.

2. The robot automatic grasping path planning method based on vision recognition according to claim 1, characterized in that, The generation of a three-dimensional dense reconstruction model of the scene using a multimodal image fusion algorithm includes: Perform white balance correction and Gaussian filtering noise reduction on RGB images, and extract color feature vectors. Hole filling and outlier removal are performed on the depth image to generate a depth feature matrix. Voxel filtering downsampling is performed on the original point cloud data to preserve geometric feature points; a multimodal fusion criterion function is constructed. ,in The registration loss between the RGB image and the depth image is... The feature matching loss between RGB images and point cloud data, To minimize the geometric consistency loss between depth images and point cloud data, the fusion criterion function is minimized using gradient descent. To optimize the registration accuracy of multimodal data, a three-dimensional dense mesh model of the scene is generated using the Poisson reconstruction algorithm.

3. The robot automatic grasping path planning method based on vision recognition according to claim 1, characterized in that, The three-dimensional dense reconstruction model performs target detection and feature extraction, identifying the three-dimensional contour and surface curvature distribution of the target object, including: A point cloud-based YOLOv8 model is used to perform target detection on a 3D dense reconstruction model, and the bounding box coordinates and category labels of the target objects are output. The target object point cloud and the background point cloud are segmented by a region growing algorithm, and the three-dimensional contour line of the target object is extracted. The sampling interval of the contour line is a preset small length. Calculate the maximum principal curvature of each vertex on the surface of the target object. and minimum principal curvature Through formula Calculate the curvature difference coefficient to generate a heat map of the curvature distribution on the surface of the target object.

4. The robot automatic grasping path planning method based on vision recognition according to claim 3, characterized in that, The three-dimensional contour and surface curvature distribution determine the candidate region for grasping, including: Calculate the minimum bounding cube of the target object based on the 3D contour lines to determine the object's spatial orientation and center coordinates; Set curvature smoothness threshold Through formula Regions that meet certain criteria are selected as candidate smooth regions, where the smoothness threshold is... Based on the clamping stability requirements of the end effector, this formula directly eliminates edges and sharp corners with abrupt curvature changes. Perform area screening on candidate flat areas and set an area threshold. The area threshold This is a preset multiple of the clamping area of ​​the end effector. The actual area of ​​the candidate flat region is determined by the formula. Filter regions that meet the criteria, and use each region that meets the criteria as a candidate region for capture. Record the center coordinates, normal vector, and extension range of each candidate region.

5. The method for automatic robot grasping path planning based on visual recognition according to claim 1, characterized in that, The selection of optimal grasping points through visual attention mechanisms includes: Constructing a visual attention evaluation function ,in This is a comprehensive evaluation score for visual attention. As a curvature smoothing evaluation factor, As the evaluation factor for the central location, As a visual visibility evaluation factor; Through the formula: ,in The curvature attenuation coefficient, The curvature difference coefficient. This represents the straight-line distance from the center of the candidate region to the center of the object. The maximum size of the object. The occlusion rate of the candidate region from the camera's perspective; Evaluation function based on visual attention The candidate regions are sorted in descending order of function value. A preset number of regions with the highest function value ranking are selected, and the geometric center point of each region is used as the candidate capture point. The three-dimensional coordinates and normal vector of the candidate capture point are output.

6. The robot automatic grasping path planning method based on vision recognition according to claim 1, characterized in that, The construction of the dynamic environmental obstacle probability map includes: The target scene space is divided into a uniform three-dimensional voxel grid, and the voxel size is consistent with the minimum motion step of the robot end effector. The occupancy status of each voxel grid is determined based on multi-view visual data. The occupancy status includes free state, occupied state and unknown state. Occupied voxels are assigned initial high probability values, free voxels are assigned initial low probability values, and unknown voxels are assigned initial intermediate probability values. The occupancy probability of each voxel is updated using Bayesian filtering, as shown in the formula: ,in The voxel occupancy probability of the current frame. This represents the voxel occupancy probability of the previous frame. The conditional probability of observing an obstacle in the occupied state. Given the conditional probability of observing obstacles in a free state, generate a dynamic environment obstacle probability map.

7. The method for automatic robot grasping path planning based on visual recognition according to claim 6, characterized in that, The temporal visual tracking algorithm is used to update the obstacle position and movement trend, and the interference weight of the obstacle on the grasping path is quantified as follows: The Kalman filter algorithm is used to perform temporal tracking on obstacle objects with an occupancy probability greater than a preset threshold, and an obstacle motion state model is established. The formula is: ,in This is the state vector representing the position and velocity of the obstacle in the current frame. Here is the state transition matrix. This is the obstacle state vector from the previous frame. To control the input matrix, For control vectors, This is the process noise vector; Based on observation model Update obstacle status, where This is the visual observation vector for the current frame. For the observation matrix, To observe the noise vector, predict the location distribution of obstacles within a preset number of frames in the future; Through formula Calculate the interference weights, where The interference weight of obstacles on the path, The shortest straight-line distance between the obstacle and the planned path. The distance attenuation coefficient is used in the path cost function to quantify the degree of influence of obstacles on path feasibility.

8. The robot automatic grasping path planning method based on vision recognition according to claim 1, characterized in that, The initial crawling path planning based on the improved fast expanding random tree algorithm includes: Let the robot's initial posture be the starting point and the optimal grasping point be the target point. Construct the state space of the robot's joint space. ,in This is the joint space state vector. The number of robot joints. For the first The angle of each joint; An improved sampling strategy for the fast expanding random tree algorithm is proposed, which sets a sampling probability bias coefficient near the target point to increase the sampling density in the target area. Define path cost function ,in The total cost of the path. , , These are the cost coefficients, The spatial length of the path. This is the sum of the interference weights of all obstacles along the path. Energy consumption for robot joint movement; The path tree generated by the greedy algorithm is pruned and optimized, and the path with the lowest cost is retained as the initial crawling path, while the number of path nodes is controlled within a preset range.

9. The robot automatic grasping path planning method based on vision recognition according to claim 8, characterized in that, The introduction of path smoothness constraints and robot joint motion limit parameters, and the optimization of path nodes using Bézier curves, includes: Set the angle limits, angular velocity limits, and angular acceleration limits for each joint of the robot as hard constraints for path optimization; Define the path smoothness constraint, as shown in the formula: ,in As an evaluation index for path smoothness, For path in parameters The second derivative at that point, For the parameters of the Bézier curve, minimize To ensure a smooth path; The nodes of the initial crawling path are fitted using a cubic Bézier curve. The formula for the Bézier curve is: ,in For parameters The corresponding path point coordinates, The coordinates of the control nodes of the path. To control the number of nodes, These are Bessel basis functions; The control points of the Bézier curve are optimized using the gradient descent algorithm to ensure that the optimized path satisfies the joint motion limit constraints and smoothness constraints.

10. The robot automatic grasping path planning method based on vision recognition according to claim 1, characterized in that, The fusion of visual servo feedback and path deviation prediction, and the real-time correction of path parameters, include: The robot uses a vision sensor at its end effector to collect image data in real time during the grasping process and calculates the positional deviation between the end effector and the optimal grasping point. and attitude deviation ,in This is a three-dimensional position deviation vector. This is a three-dimensional attitude deviation vector; A path deviation prediction model is constructed, which uses a long short-term memory network to learn historical deviation sequences and outputs the predicted deviation value for the future preset control period. and ,in This is the position deviation prediction vector. This is the attitude deviation prediction vector; Through formula Define the correction amount, where This is the joint angle correction vector. The pseudo-inverse of the Jacobian matrix for the robot. For predicting weighting coefficients; Correction amount The joint angle parameters are superimposed on the original path, and the path parameters are updated at preset time intervals to achieve real-time closed-loop correction of the grasping process.

Images