Manipulator target grabbing path intelligent planning method based on visual depth perception

Abstract

The invention discloses a manipulator target grabbing path intelligent planning method based on visual depth perception. The manipulator target grabbing path intelligent planning method comprises the following steps that grabbing target point cloud data are obtained and preprocessed to generate a structured point cloud data set; executing SuperPoint division and adjacency relation construction, and generating a SuperPoint graph structure; constructing a plurality of groups of action guide graphs and embedding the action guide graphs into an SPGNet attention mechanism to form an improved SPGNet structure; inputting structured point cloud data, extracting node features and capturing response, and generating a capturing candidate region and a sorting result; determining grabbing point coordinates and grabbing posture parameters according to the sorting result; path search and obstacle prediction are executed in combination with the tail end pose of the mechanical arm, and grabbing path trajectory data are generated; new point clouds are collected in the grabbing process, a structure topology self-adaptive updating unit is called to reconstruct a graph structure, and a final control instruction is output. According to the invention, high-precision grabbing and path self-adaptive control of the manipulator in a complex environment are realized, and the dynamic operation stability is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of path intelligent control, and in particular to a method for intelligent planning of a manipulator target grasping path based on visual depth perception. Background Art

[0002] In industrial automation and intelligent manufacturing scenarios, robotic arms have been widely used in tasks such as object handling, assembly, and sorting. To improve operational autonomy and environmental adaptability, some existing methods introduce visual sensors to obtain target images and depth information, locate the target position through image processing algorithms, and generate grasping actions in combination with path planning algorithms. However, such methods usually rely on static scene assumptions. In dynamically changing or complex working environments, target recognition accuracy decreases, path planning robustness is insufficient, and the grasping success rate is difficult to guarantee. At the same time, existing methods have limited structural modeling capabilities for three-dimensional point clouds, making it difficult to accurately express the boundary information and geometric features of target objects, affecting the stability judgment of the grasping points.

[0003] Some technical solutions introduce graph neural networks to perform structured modeling of point cloud data, but most models lack a guidance mechanism that is semantically associated with the grasping task and are unable to adjust the extraction direction of graph structure features according to specific action strategies. In addition, preset obstacle maps or simplified models are often used in the path generation process, and there is a lack of real-time topology perception and path reconstruction mechanisms, resulting in an inability to effectively respond to target movement or obstacle interference during execution. Especially in grasping tasks, there is a strong correlation between the path trajectory and the target contact area. The lack of dynamic adjustment capability in path planning will directly affect the accuracy and safety of the final grasping control.

[0004] The existing technology lacks a unified grasping path planning method that can integrate visual depth perception, dynamic updating of graph structure and action guidance mechanism, and cannot meet the comprehensive requirements of target recognition accuracy, adaptive path planning and dynamic control strategy in complex scenarios.

[0005] Therefore, how to provide an intelligent planning method for the robot target grasping path based on visual depth perception is an urgent problem that needs to be solved by those skilled in the art. Summary of the Invention

[0006] One purpose of the present invention is to propose an intelligent planning method for the path of a manipulator target grasping based on visual depth perception. The present invention integrates point cloud SuperPoint structural modeling, an improved SPGNet perception network and an action guidance mechanism, and combines structural topology adaptive updating and path reconstruction strategies to achieve stable grasping in dynamic scenarios. It has the advantages of high recognition accuracy, strong path adaptability and high real-time control response.

[0007] According to an embodiment of the present invention, a method for intelligently planning a manipulator target grasping path based on visual depth perception includes the following steps:

[0008] Obtain the point cloud data of the grasped target, pre-process and construct a structured point cloud data set; perform SuperPoint partitioning operation on the structured point cloud data set, generate several SuperPoint areas based on the geometric structure and density features, construct the connecting edges between SuperPoints based on the Euclidean space adjacency relationship, and generate a SuperPoint graph structure; construct several action guidance graphs based on the gripper structure and target direction, input the action guidance graph into the attention mechanism of SPGNet, fuse the SuperPoint graph structure and the action guidance graph, and form an improved SPGNet structure based on the multi-scale action guidance graph attention mechanism; input the structured point cloud data set into the improved SPGNet structure, extract SuperPoint The spatial position features, boundary response features and grasping candidate response values ​​of t are combined to generate a grasping candidate area set and grasping priority sorting results; based on the grasping candidate area set and grasping priority sorting results, the spatial three-dimensional coordinates of the target grasping point and the corresponding grasping posture parameters are determined; according to the current posture of the end of the robot arm and the spatial three-dimensional coordinates of the target grasping point, a path search task is constructed, and path planning operations based on graph search and obstacle prediction are performed to generate optimal grasping path trajectory data; during the process of the robot arm executing the grasping path, a new round of structured point cloud data set is constructed, the structural topology adaptive update unit is called to correct the SuperPoint graph structure, and the grasping candidate area set and grasping path trajectory data are re-evaluated based on the updated SPGNet structure to generate the final grasping control instructions.

[0009] Optionally, it is characterized in that the process of acquiring the point cloud data includes: spatially aligning the RGB image and depth image of the target grasped by the robot to generate point cloud data; the preprocessing process includes: performing voxel filtering and boundary cropping on the point cloud data.

[0010] Optionally, the process of performing the SuperPoint partitioning operation to generate a plurality of SuperPoint regions according to geometric structure and density features includes:

[0011] Calculate the curvature value for each point in the structured point cloud data set , normal vector direction , and the local point density value ; Points that meet the following conditions are divided into the same initial cluster unit: the curvature difference satisfies , the normal vector angle satisfies ,radius The difference in the number of inner neighbor points satisfies ;in, 、 、 They are the preset curvature difference threshold, normal vector angle threshold, and neighbor point number difference threshold, , , Subscript , Indicates correspondence Point and The curvature value, normal vector direction, and local point density value of the point are calculated; the connectivity judgment based on Euclidean distance is performed on the initial clustering units, and the discrete point clusters whose number of points in the connected domain is less than the threshold are eliminated; the remaining clustering units are subjected to boundary merging processing, overlapping boundary points are deleted, and the boundary contours are smoothed; the number of cluster points is screened satisfy , and the clustering units whose Euclidean distance between the farthest points does not exceed the threshold are output as SuperPoint area sets.

[0012] Optionally, it is characterized in that the generation of the SuperPoint graph structure includes:

[0013] For the SuperPoint area set, extract the centroid coordinates of each SuperPoint, calculate the centroid Euclidean distance between any two SuperPoints, and determine whether it is less than the set adjacency threshold. Construct the SuperPoint pairs whose distance does not exceed the adjacency threshold into an undirected connection edge, and generate the graph structure for the constructed undirected edge set to form the initial SuperPoint graph structure; perform connectivity detection on the initial SuperPoint graph structure. If there are more than two connected subgraphs, select the SuperPoint pair with the smallest distance between the centroids between the subgraphs, and supplement the connection edges until the entire graph structure forms a single connected graph as the final input SuperPoint graph structure.

[0014] Optionally, it is characterized in that the step of constructing an improved SPGNet structure based on a multi-scale action-guided graph attention mechanism includes:

[0015] Based on the centroid coordinates, normal vector direction and neighborhood point distribution characteristics of each SuperPoint node in the SuperPoint graph structure, combined with the preset grasping direction encoding set, several grasping direction candidate perspectives are constructed, and each candidate perspective generates a set of action guidance graphs; each action guidance graph is aligned according to the order of SuperPoint nodes to generate a one-dimensional floating-point response vector, which reflects the response strength of each SuperPoint node in the current spatial structure under the grasping direction perspective; a scale resampling operation is performed on each set of action guidance graphs to generate a multi-scale response vector set, which is input into the graph convolution structure; in the edge feature update stage of SPGNet, for any connecting node With node Edge, extract nodes With node The response value in the response vector of the action guidance graph at each scale is recorded as and , calculate the weighted mean as the edge The guided contribution value under the scale is weighted Weighted fusion to obtain the guidance coefficient ; The guide coefficient With the current edge feature vector Perform channel-dimensional weighted fusion to generate updated edge feature vectors ,in, is the initial edge feature vector; in the node aggregation stage, for each SuperPoint node, the updated edge feature vector set of all adjacent edges is collected, the values ​​of the node in the action guidance graph response vector of all scales are fused, and maximum pooling and mean fusion are performed to generate the node attention weight vector; the node attention weight vector and the node current feature vector are weightedly calculated channel by channel, and the SuperPoint node features after the fusion of action guidance information are output as the input for grasping area recognition and priority sorting.

[0016] Optionally, it is characterized in that the structured point cloud data set is input into the improved SPGNet structure, the spatial position features, boundary response features and grasp candidate response values ​​of SuperPoint are extracted, and a grasp candidate area set and grasp priority sorting results are generated, specifically including:

[0017] Input the structured point cloud data set into the multi-layer graph convolutional network of the improved SPGNet structure;

[0018] In each graph convolution layer, graph neural transmission and nonlinear transformation are performed based on the SuperPoint node features after integrating the action guidance graph information, and a node feature set is output; in the last layer of graph convolution output, the following three types of output vectors are calculated for each SuperPoint node: a spatial position feature vector, which represents the relative position relationship of the SuperPoint center of mass coordinates in the global space; a boundary response feature vector, which calculates the degree of boundary mutation based on the difference in node normal vectors and local curvature changes; and a grasp candidate response value, which represents the feasibility score of the SuperPoint area as a grasp contact area; threshold screening is performed based on the grasp candidate response values ​​of all SuperPoint nodes, and a set of nodes with response values ​​higher than the set threshold are extracted as a grasp candidate area set; for each SuperPoint node in the grasp candidate area set, a candidate area scoring vector is constructed, which is composed of a linearly weighted combination of the grasp candidate response value, the boundary response feature amplitude and the spatial position feature; according to the weighted score result of the scoring vector, all grasp candidate areas are arranged in descending order according to the score value to generate a grasp priority ranking result. Optionally, it is characterized in that the grasping posture parameters are composed of a grasping normal vector, a gripper rotation angle and a contact surface orientation, and are calculated based on the normal vector direction and neighborhood structure of the SuperPoint where the target grasping point is located.

[0019] Optionally, it is characterized in that the generation of the optimal grasping path trajectory data includes:

[0020] A six-dimensional path state space is constructed based on the current position of the robot end and the three-dimensional coordinates of the target grasping point. The path search graph is constructed within the six-dimensional path state space by combining the obstacle boundary information in the SuperPoint graph structure. The inaccessible areas in the search graph are updated using the obstacle motion trend prediction results. The graph search algorithm is executed to generate a path trajectory sequence, which is then smoothed to generate the optimal grasping path trajectory data.

[0021] Optionally, it is characterized in that the structure of the structural topology adaptive updating unit includes:

[0022] The update module is used to collect new RGB images and depth images in real time during the execution of the path by the robotic arm to generate the current frame structured point cloud data set; the reconstruction module is used to recalculate the center of mass position, normal vector direction and adjacent edge set of the SuperPoint node based on the current frame structured point cloud data set; the matching module is used to match the node correspondence between the SuperPoint graph structure of the current frame and the graph structure of the previous frame, and identify node position drift and connection edge changes; the correction module is used to add, delete and adjust the weight of the connection edges in the SuperPoint graph structure according to the matching results to generate an updated SuperPoint graph structure.

[0023] Optionally, the process of generating the final crawl control instruction includes the following strategy update mechanism:

[0024] When the structural topology adaptive update unit detects that the center of mass position offset of the node where the grasping target is located in the SuperPoint graph structure in two consecutive frames of structured point cloud exceeds the set threshold, the grasping candidate area set is re-evaluated, the current target grasping point is replaced and the grasping posture parameters are updated; when any path segment in the optimal grasping path trajectory is judged to be an obstacle area in the corresponding position of the new frame of structured point cloud, the path search task is re-executed to generate new optimal grasping path trajectory data; when the action guidance graph response value has a score mutation in the grasping candidate area and the response value decreases by more than the set threshold, the improved SPGNet structure is called to recalculate the SuperPoint node features and re-arrange the grasping priority sorting results; when any strategy is triggered, the final grasping control instruction is recombined based on the updated target grasping point, grasping posture parameters and path trajectory data, and sent to the robot arm control end to execute the adjusted grasping action.

[0025] The beneficial effects of the present invention are:

[0026] (1) Enhance the ability to recognize three-dimensional target structures: By performing SuperPoint partitioning and graph structure construction on point cloud data, the improved SPGNet structure is used to extract spatial position features, boundary response features and grasp candidate response values, effectively improving the recognition accuracy of target grasping areas under complex geometric structures and avoiding grasping failures caused by blurred boundaries or irregular shapes.

[0027] (2) Realize dynamic adaptive optimization of grasping paths: Construct path search tasks based on graph search and obstacle prediction, combine the structure topology adaptive update unit during path execution, update the SuperPoint graph structure and obstacle area information in real time, support path replanning and grasping strategy adjustment, and ensure stable grasping effects in dynamic interference or target movement scenarios.

[0028] (3) Introducing a multi-scale action guidance mechanism to strengthen the semantic focus of grasping: Using multi-angle grasping perspectives to construct an action guidance graph and embed it into the SPGNet edge feature and node aggregation process, the model's perception selectivity of the operation target area is enhanced, the semantic consistency from perception to action is improved, and the grasping success rate and execution accuracy are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0030] Figure 1 This is a flow chart of the intelligent planning method for manipulator target grasping path based on visual depth perception proposed by the present invention;

[0031] Figure 2 This is a flowchart of structured point cloud data processing and SuperPoint graph structure construction for the intelligent planning method of manipulator target grasping path based on visual depth perception proposed in the present invention;

[0032] Figure 3 This is a schematic diagram of the improved SPGNet structure of the intelligent planning method for manipulator target grasping path based on visual depth perception proposed in this invention. DETAILED DESCRIPTION

[0033] The present invention will now be described in further detail with reference to the accompanying drawings, which are simplified schematic diagrams that illustrate the basic structure of the present invention in a schematic manner.

[0034] refer to Figure 1-3 , an intelligent planning method for a manipulator target grasping path based on visual depth perception, comprising the following steps:

[0035] Obtain the point cloud data of the grasped target, pre-process and construct a structured point cloud data set; perform SuperPoint partitioning operation on the structured point cloud data set, generate several SuperPoint areas based on the geometric structure and density features, construct the connecting edges between SuperPoints based on the Euclidean space adjacency relationship, and generate a SuperPoint graph structure; construct several action guidance graphs based on the gripper structure and target direction, input the action guidance graph into the attention mechanism of SPGNet, fuse the SuperPoint graph structure and the action guidance graph, and form an improved SPGNet structure based on the multi-scale action guidance graph attention mechanism; input the structured point cloud data set into the improved SPGNet structure, extract SuperPoint The spatial position features, boundary response features and grasping candidate response values ​​of t are combined to generate a grasping candidate area set and grasping priority sorting results; based on the grasping candidate area set and grasping priority sorting results, the spatial three-dimensional coordinates of the target grasping point and the corresponding grasping posture parameters are determined; according to the current posture of the end of the robot arm and the spatial three-dimensional coordinates of the target grasping point, a path search task is constructed, and path planning operations based on graph search and obstacle prediction are performed to generate optimal grasping path trajectory data; during the process of the robot arm executing the grasping path, a new round of structured point cloud data set is constructed, the structural topology adaptive update unit is called to correct the SuperPoint graph structure, and the grasping candidate area set and grasping path trajectory data are re-evaluated based on the updated SPGNet structure to generate the final grasping control instructions.

[0036] In this embodiment, it is characterized in that the process of acquiring the point cloud data includes: spatially aligning the RGB image and the depth image of the target grasped by the robot to generate point cloud data; the preprocessing process includes: performing voxel filtering and boundary cropping on the point cloud data.

[0037] In this embodiment, the process of performing the SuperPoint partitioning operation to generate a plurality of SuperPoint regions according to geometric structure and density characteristics includes:

[0038] Calculate the curvature value for each point in the structured point cloud data set , normal vector direction , and the local point density value ; Points that meet the following conditions are divided into the same initial cluster unit: the curvature difference satisfies , the normal vector angle satisfies ,radius The difference in the number of inner neighbor points satisfies ;in, 、 、 They are the preset curvature difference threshold, normal vector angle threshold, and neighbor point number difference threshold, , , Subscript , Indicates correspondence Point and The curvature value, normal vector direction, and local point density value of the point are determined; the connectivity judgment based on Euclidean distance is performed on the initial clustering unit, and the discrete point clusters whose number of points in the connected domain is less than the threshold are eliminated; specifically, the following steps are performed: constructing an adjacency graph for each initial clustering unit, establishing an undirected edge when the Euclidean distance between any two points does not exceed the preset connection distance threshold, performing a connected domain search operation in the adjacency graph, extracting all connected subgraphs, and counting the number of points in each connected subgraph. If the number of points in a connected subgraph is less than the minimum connected point threshold, all points corresponding to the subgraph are eliminated from the clustering unit, and the point set consisting of the remaining connected subgraphs is retained as the updated clustering result;

[0039] The remaining cluster units perform boundary merging processing, delete overlapping boundary points and smooth the boundary contour; specifically: perform boundary extraction operation on each remaining cluster unit, judge the cluster boundary point based on the Euclidean distance and normal vector angle between the cluster internal point and its adjacent cluster, perform spatial overlap detection on the boundary point set of all cluster units, if the distance between the boundary points of different clusters in the three-dimensional coordinate space is less than the preset overlap distance threshold, it is determined to be an overlapping boundary point, delete one side's redundant boundary point on the basis of retaining the continuity of the contour, perform contour smoothing processing on the merged boundary point set, use B-spline fitting to adjust the local mutation position in the boundary point sequence, generate a smooth and continuous boundary contour as the final SuperPoint area boundary output; filter the number of cluster points satisfy , and the clustering units whose Euclidean distance between the farthest points does not exceed the threshold are output as SuperPoint area sets.

[0040] In this embodiment, the generation of the SuperPoint graph structure includes:

[0041] For the SuperPoint area set, extract the centroid coordinates of each SuperPoint, calculate the centroid Euclidean distance between any two SuperPoints, and determine whether it is less than the set adjacency threshold. Construct the SuperPoint pairs whose distance does not exceed the adjacency threshold into an undirected connection edge, and generate the graph structure for the constructed undirected edge set to form the initial SuperPoint graph structure; perform connectivity detection on the initial SuperPoint graph structure. If there are more than two connected subgraphs, select the SuperPoint pair with the smallest distance between the centroids between the subgraphs, and supplement the connection edges until the entire graph structure forms a single connected graph as the final input SuperPoint graph structure.

[0042] In this embodiment, it is characterized in that the step of constructing an improved SPGNet structure based on a multi-scale action-guided graph attention mechanism includes:

[0043] Based on the centroid coordinates, normal vector direction and neighborhood point distribution characteristics of each SuperPoint node in the SuperPoint graph structure, combined with the preset grasping direction encoding set, several grasping direction candidate perspectives are constructed, and each candidate perspective generates a set of action guidance graphs; each action guidance graph is aligned according to the order of SuperPoint nodes to generate a one-dimensional floating-point response vector, which reflects the response strength of each SuperPoint node in the current spatial structure under the grasping direction perspective; a scale resampling operation is performed on each set of action guidance graphs to generate a multi-scale response vector set, which is input into the graph convolution structure; specifically, each set of action guidance graphs is resampled with SuperPoint nodes to generate a multi-scale response vector set, which is input into the graph convolution structure; specifically, The oint node constructs the initial response value sequence for the alignment benchmark, and uses the Gaussian kernel blur function to perform convolution smoothing on the response value sequence. Based on the convolution smoothing result, the initial guidance graph is subjected to pyramid resampling operation according to the set scale layer and downsampling ratio, and action guidance graph subsequences at different resolutions are generated in sequence. In the action guidance graph at each scale, the SuperPoint node index consistency is maintained to ensure that the multi-scale graph structure has the channel dimension alignment condition in the process of inputting the graph neural network, and the guidance graphs of each scale are output as a set of floating-point feature vectors of the same dimension as the attention modulation input in the graph convolution structure; in the edge feature update stage of SPGNet, for any connecting node With node Edge, extract nodes With node The response value in the response vector of the action guidance graph at each scale is recorded as and , calculate the weighted mean as the edge The guided contribution value under the scale is weighted Weighted fusion to obtain the guidance coefficient ; The guide coefficient With the current edge feature vector Perform channel-dimensional weighted fusion to generate updated edge feature vectors ,in, is the initial edge feature vector; in the node aggregation stage, for each SuperPoint node, the updated edge feature vector set of all adjacent edges is collected, the values ​​of the node in the action guidance graph response vector of all scales are fused, and the maximum pooling and mean fusion are performed to generate the node attention weight vector; specifically: construct a numerical vector for the input action guidance response value sequence , where each Indicates that the SuperPoint node is in The response value of the bootstrap map at each scale is the vector Perform a maximum pooling operation to extract the maximum value of all scale response values , used to capture extreme value information in the scale dimension, for vector At the same time, the mean fusion operation is performed to calculate the average value of all scale response values , used to reflect the overall response trend of the scale, and The combined result is used as the scale fusion result and input into the subsequent node attention weight construction process;

[0044] The node attention weight vector and the node current feature vector are weightedly calculated channel by channel, and the SuperPoint node feature after integrating the action guidance information is output as the input for grasping area recognition and priority sorting.

[0045] This embodiment is characterized in that the structured point cloud data set is input into the improved SPGNet structure, the spatial position features, boundary response features and grasp candidate response values ​​of SuperPoint are extracted, and a grasp candidate area set and grasp priority ranking results are generated, specifically including:

[0046] Input the structured point cloud data set into the multi-layer graph convolutional network of the improved SPGNet structure;

[0047] In each graph convolution layer, graph neural transmission and nonlinear transformation are performed based on the SuperPoint node features after integrating the action guidance graph information, and a node feature set is output; specifically: in each graph convolution layer, the feature representation of the current SuperPoint node is updated, and all adjacent nodes of the node are traversed first, and the feature vectors of the adjacent nodes and the feature vectors of the connecting edges are weightedly aggregated according to the set weights to form an aggregated feature input. During the aggregation process, the structural correlation between adjacent nodes is retained, and the direction selectivity is enhanced by combining the guidance coefficient in the edge feature. After the aggregation is completed, a nonlinear transformation operation is performed on the aggregated feature input. The nonlinear transformation uses an activation function with a threshold segmentation characteristic to map the linear combination result to a nonlinear output to improve the distinguishing ability of the feature expression. The final output node feature vector is used for the output of the graph convolution layer and as the input basis for the next layer of graph convolution operation; in the last layer of graph convolution output, The following three types of output vectors are calculated for each SuperPoint node: a spatial position feature vector, which represents the relative position relationship of the SuperPoint centroid coordinates in global space; a boundary response feature vector, which calculates the degree of boundary mutation based on the difference in node normal vectors and local curvature changes; and a grasp candidate response value, which represents the feasibility score of the SuperPoint area as a grasp contact area; threshold screening is performed based on the grasp candidate response values ​​of all SuperPoint nodes, and a set of nodes with response values ​​higher than the set threshold is extracted as a grasp candidate area set; for each SuperPoint node in the grasp candidate area set, a candidate area scoring vector is constructed, which is composed of a linear weighted combination of the grasp candidate response value, the boundary response feature amplitude, and the spatial position feature; based on the weighted score result of the scoring vector, all grasp candidate areas are sorted in descending order according to the score value to generate a grasp priority ranking result. In this embodiment, it is characterized in that the grasping posture parameters are composed of the grasping normal vector, the gripper rotation angle, and the contact surface orientation, and are calculated based on the normal vector direction and neighborhood structure of the SuperPoint where the target grasping point is located.

[0048] In this embodiment, the generation of the optimal grasping path trajectory data includes:

[0049] A six-dimensional path state space is constructed based on the current position of the robot end and the three-dimensional coordinates of the target grasping point. The path search graph is constructed within the six-dimensional path state space by combining the obstacle boundary information in the SuperPoint graph structure. The inaccessible areas in the search graph are updated using the obstacle motion trend prediction results. The graph search algorithm is executed to generate a path trajectory sequence, which is then smoothed to generate the optimal grasping path trajectory data.

[0050] In this embodiment, the structure of the structure topology adaptive updating unit includes:

[0051] The update module is used to collect new RGB images and depth images in real time during the execution of the path by the robotic arm to generate the current frame structured point cloud data set; the reconstruction module is used to recalculate the center of mass position, normal vector direction and adjacent edge set of the SuperPoint node based on the current frame structured point cloud data set; the matching module is used to match the node correspondence between the SuperPoint graph structure of the current frame and the graph structure of the previous frame, and identify node position drift and connection edge changes; the correction module is used to add, delete and adjust the weight of the connection edges in the SuperPoint graph structure according to the matching results to generate an updated SuperPoint graph structure.

[0052] In this embodiment, the feature is that the process of generating the final grab control instruction includes the following strategy update mechanism:

[0053] When the structural topology adaptive update unit detects that the center of mass position offset of the node where the grasping target is located in the SuperPoint graph structure in two consecutive frames of structured point cloud exceeds the set threshold, the grasping candidate area set is re-evaluated, the current target grasping point is replaced and the grasping posture parameters are updated; when any path segment in the optimal grasping path trajectory is judged to be an obstacle area in the corresponding position of the new frame of structured point cloud, the path search task is re-executed to generate new optimal grasping path trajectory data; when the action guidance graph response value has a score mutation in the grasping candidate area and the response value decreases by more than the set threshold, the improved SPGNet structure is called to recalculate the SuperPoint node features and re-arrange the grasping priority sorting results; when any strategy is triggered, the final grasping control instruction is recombined based on the updated target grasping point, grasping posture parameters and path trajectory data, and sent to the robot arm control end to execute the adjusted grasping action.

[0054] Example 1:

[0055] In order to verify the feasibility of the present invention in implementation, the present invention is applied to a certain industrial automatic grasping assembly line environment. The robot in this scenario needs to identify, plan paths and perform precise grasping operations on multiple types of workpieces randomly placed on the assembly line. Due to the irregular shape of the workpieces, the complex background environment, and the presence of factors such as occlusion, lighting changes and dynamic entry of interference objects, the traditional grasping method based on single-frame RGB image recognition and fixed path generation often has problems such as misjudgment of position, path collision or clamping failure. The robot's grasping success rate is less than 75%, and it cannot respond in real time in interference scenarios, seriously affecting the stability of the operation rhythm.

[0056] In this application scenario, an industrial robotic arm with an RGB-D sensor is deployed to collect RGB images and depth images in real time for each grasping task, and construct structured point cloud data. The point cloud data is divided into multiple structurally continuous regions through the SuperPoint partitioning algorithm, and a SuperPoint graph structure is constructed for graph neural network processing. Based on the improved SPGNet model, the system performs feature extraction and candidate grasping area identification on the SuperPoint node, generates multiple grasping action guidance maps in each operation cycle, and integrates multi-scale information to guide the network attention distribution, and finally outputs the grasping candidate area set and the grasping priority scoring results.

[0057] In each round of operation, the system generates the grasping point coordinates and posture parameters based on the SuperPoint node with the highest grasping priority, and combines the real-time position of the current end effector to construct a six-dimensional path search space, and executes graph search path planning in the path graph. If there is occlusion or the grasping target position drifts during the grasping process, the system automatically calls the structural topology adaptive update unit to reconstruct the SuperPoint graph structure, re-executes path planning and grasping point evaluation based on the updated structure, and finally outputs control instructions to drive the robotic arm to complete the operation.

[0058] The test selected 10 typical workpiece categories, and tested 100 rounds of grasping tasks for each category, comparing the performance of the method of the present invention with the traditional method based on fixed boundary extraction and heuristic path strategy. The results show that in interference-free scenarios, the average recognition accuracy of the method of the present invention is 97.3%, and the grasping success rate is 95.6%; in dynamic occlusion scenarios, the success rate of the traditional method dropped to 64.8%, while the method of the present invention remained at 90.2%, and the path reconstruction response time was controlled within 0.53 seconds, ensuring the stable operation rhythm of the system. In the grasping posture evaluation, the average clamping deviation angle of the method of the present invention was 2.1 degrees, while that of the traditional method was 5.4 degrees, which improved the clamping stability and positioning accuracy.

[0059] The present invention demonstrates a higher target grasping success rate, faster path response capability and smaller control error in typical complex scenarios, verifying the practicality and superiority of the fusion of SuperPoint graph modeling, multi-scale action-guided attention mechanism and structure adaptive update strategy in dynamic manipulation tasks.

[0060] Table 1: Comparative data of typical workpiece grasping performance

[0061] Workpiece number Type Shape Crawl method Average recognition accuracy (%) Average crawling success rate (%) Average clamping deviation angle (°) Success rate under dynamic occlusion (%) Average path reconstruction response time (s) 01 Special-shaped blocks Surface adhesion adsorption 96.8 94.1 2.3 89.5 0.51 02 Tubular parts Inner wall clamping 98.2 96.4 1.7 91.0 0.49 03 Short column workpiece Clamping with claws on both sides 97.5 95.7 2.0 90.3 0.53 04 Saw gear Single tooth alignment clamping 97.9 96.0 2.2 91.6 0.55 05 Spherical components Top Adsorption 96.1 94.6 2.5 88.7 0.54 06 Concave structural parts Inner wall support and clamping 97.6 95.3 2.0 89.9 0.52 07 Plate slider Top negative pressure adsorption 98.3 96.2 1.9 91.2 0.50 08 Irregular metal blocks Adaptive surface fitting 97.0 94.8 2.6 89.1 0.56 09 Cylindrical sleeve Three-point positioning of the gripper 97.8 95.9 1.8 90.7 0.51 10 flat disc Central hole adsorption 96.9 94.4 2.4 88.9 0.55

[0062] The above table shows the performance of the method of the present invention in different types of workpiece grasping tasks, and makes a quantitative comparison with existing traditional methods. The test content covers core indicators such as workpiece recognition accuracy, grasping success rate, clamping deviation, dynamic occlusion adaptability and path reconstruction response speed. Through the analysis of measured data of ten typical workpieces, it can be clearly seen that the present invention has significant advantages in many key performance aspects.

[0063] Recognition accuracy remained above 96% for all workpiece types, with recognition accuracy exceeding 98% for regular structures such as tubular parts and plate-shaped sliders. This demonstrates the versatility and stability of the processing approach based on SuperPoint partitioning and the SPGNet architecture. The average grasping success rate exceeded 95%, with success rates exceeding 96% for workpieces with clear structures and stable gripping, such as sawtooth wheels, cylindrical sleeves, and short cylindrical workpieces. This demonstrates that the multi-scale guidance mechanism of the action guidance graph can effectively improve the accuracy of target area positioning and posture matching.

[0064] Under dynamic occlusion conditions, the present invention demonstrates better robustness than traditional methods, with an average success rate maintained at over 90%. Compared with the traditional method's performance of around 64% in similar scenarios, it effectively improves the operational reliability of the robot in non-ideal environments. The path reconstruction response time is all controlled within 0.56 seconds, and the shortest can reach 0.49 seconds, reflecting the linkage efficiency between the structural topology adaptive update unit and the path search module, meeting the continuous operation requirements of the production rhythm.

[0065] In terms of the clamping deviation angle, the method of the present invention controls it between 1.7° and 2.6°, which is significantly better than the average error range of 5.4° of the traditional method. Especially in workpieces such as tubular parts, plate sliders, and cylindrical sleeves, this method provides higher clamping stability. This result verifies the effectiveness of the action-guided attention mechanism and node feature fusion in grasping posture evaluation.

[0066] Overall, the method of the present invention demonstrates high recognition accuracy, strong grasping stability and excellent adaptability in high-complexity, dynamic interference and multi-modal grasping scenarios, providing stable and reliable path planning and control support for robotic arm grasping in intelligent manufacturing environments.

[0067] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. An intelligent planning method for manipulator target grasping path based on visual depth perception, characterized by: The steps include: Obtain the point cloud data of the grasped target, pre-process and construct a structured point cloud data set; perform SuperPoint partitioning operation on the structured point cloud data set, generate several SuperPoint areas based on the geometric structure and density features, construct the connecting edges between SuperPoints based on the Euclidean space adjacency relationship, and generate a SuperPoint graph structure; construct several action guidance graphs based on the gripper structure and target direction, input the action guidance graph into the attention mechanism of SPGNet, fuse the SuperPoint graph structure and the action guidance graph, and form an improved SPGNet structure based on the multi-scale action guidance graph attention mechanism; input the structured point cloud data set into the improved SPGNet structure, extract SuperPoint The spatial position features, boundary response features and grasping candidate response values ​​of t are combined to generate a grasping candidate area set and grasping priority sorting results; based on the grasping candidate area set and grasping priority sorting results, the spatial three-dimensional coordinates of the target grasping point and the corresponding grasping posture parameters are determined; according to the current posture of the end of the robot arm and the spatial three-dimensional coordinates of the target grasping point, a path search task is constructed, and path planning operations based on graph search and obstacle prediction are performed to generate optimal grasping path trajectory data; during the process of the robot arm executing the grasping path, a new round of structured point cloud data set is constructed, the structural topology adaptive update unit is called to correct the SuperPoint graph structure, and the grasping candidate area set and grasping path trajectory data are re-evaluated based on the updated SPGNet structure to generate the final grasping control instructions.

2. The method for intelligent planning of a manipulator target grasping path based on visual depth perception according to claim 1 is characterized in that: The point cloud data acquisition process includes: spatially aligning the RGB image and depth image of the target grasped by the robot arm to generate point cloud data; the preprocessing process includes: performing voxel filtering and boundary clipping on the point cloud data.

3. The method for intelligent planning of a manipulator target grasping path based on visual depth perception according to claim 2 is characterized in that: The process of performing the SuperPoint partitioning operation to generate a plurality of SuperPoint regions according to geometric structure and density characteristics includes: Calculate the curvature value for each point in the structured point cloud data set , normal vector direction , and the local point density value ; Points that meet the following conditions are divided into the same initial cluster unit: the curvature difference satisfies , the normal vector angle satisfies ,radius The difference in the number of inner neighbor points satisfies ;in, 、 、 They are the preset curvature difference threshold, normal vector angle threshold, and neighbor point number difference threshold, , , Subscript , Indicates correspondence Point and The curvature value, normal vector direction, and local point density value of the point are calculated; the connectivity judgment based on Euclidean distance is performed on the initial clustering units, and the discrete point clusters whose number of points in the connected domain is less than the threshold are eliminated; the remaining clustering units are subjected to boundary merging processing, overlapping boundary points are deleted, and the boundary contours are smoothed; the number of cluster points is screened satisfy , and the clustering units whose Euclidean distance between the farthest points does not exceed the threshold are output as SuperPoint area sets.

4. The method for intelligent planning of a manipulator target grasping path based on visual depth perception according to claim 3 is characterized in that: The generation of the SuperPoint graph structure includes: For the SuperPoint area set, extract the centroid coordinates of each SuperPoint, calculate the centroid Euclidean distance between any two SuperPoints, and determine whether it is less than the set adjacency threshold. Construct the SuperPoint pairs whose distance does not exceed the adjacency threshold into an undirected connection edge, and generate the graph structure for the constructed undirected edge set to form the initial SuperPoint graph structure; perform connectivity detection on the initial SuperPoint graph structure. If there are more than two connected subgraphs, select the SuperPoint pair with the smallest distance between the centroids between the subgraphs, and supplement the connection edges until the entire graph structure forms a single connected graph as the final input SuperPoint graph structure.

5. The method for intelligent planning of a manipulator target grasping path based on visual depth perception according to claim 4 is characterized in that: The steps of constructing the improved SPGNet structure based on the multi-scale action-guided graph attention mechanism include: Based on the centroid coordinates, normal vector direction and neighborhood point distribution characteristics of each SuperPoint node in the SuperPoint graph structure, combined with the preset grasping direction encoding set, several grasping direction candidate perspectives are constructed, and each candidate perspective generates a set of action guidance graphs; each action guidance graph is aligned according to the order of SuperPoint nodes to generate a one-dimensional floating-point response vector, which reflects the response strength of each SuperPoint node in the current spatial structure under the grasping direction perspective; a scale resampling operation is performed on each set of action guidance graphs to generate a multi-scale response vector set, which is input into the graph convolution structure; in the edge feature update stage of SPGNet, for any connecting node With node Edge, extract nodes With node The response value in the response vector of the action guidance graph at each scale is recorded as and , calculate the weighted mean as the edge The guided contribution value under the scale is weighted Weighted fusion to obtain the guidance coefficient ; The guidance coefficient With the current edge feature vector Perform channel-dimensional weighted fusion to generate updated edge feature vectors ,in, is the initial edge feature vector; in the node aggregation stage, for each SuperPoint node, the updated edge feature vector set of all adjacent edges is collected, the values ​​of the node in the action guidance graph response vector of all scales are fused, and maximum pooling and mean fusion are performed to generate the node attention weight vector; the node attention weight vector and the node current feature vector are weightedly calculated channel by channel, and the SuperPoint node features after the fusion of action guidance information are output as the input for grasping area recognition and priority sorting.

6. The method for intelligent planning of a manipulator target grasping path based on visual depth perception according to claim 5 is characterized in that: The structured point cloud data set is input into the improved SPGNet structure, the spatial position features, boundary response features and grasp candidate response values ​​of SuperPoint are extracted, and a grasp candidate area set and grasp priority ranking results are generated, specifically including: Input the structured point cloud data set into the multi-layer graph convolutional network of the improved SPGNet structure; In each graph convolution layer, graph neural transmission and nonlinear transformation are performed based on the SuperPoint node features after integrating the action guidance graph information, and a node feature set is output; in the last layer of graph convolution output, the following three types of output vectors are calculated for each SuperPoint node: a spatial position feature vector, which represents the relative position relationship of the SuperPoint center of mass coordinates in the global space; a boundary response feature vector, which calculates the degree of boundary mutation based on the difference in node normal vectors and local curvature changes; and a grasp candidate response value, which represents the feasibility score of the SuperPoint area as a grasp contact area; threshold screening is performed based on the grasp candidate response values ​​of all SuperPoint nodes, and a set of nodes with response values ​​higher than the set threshold are extracted as a grasp candidate area set; for each SuperPoint node in the grasp candidate area set, a candidate area scoring vector is constructed, which is composed of a linearly weighted combination of the grasp candidate response value, the boundary response feature amplitude and the spatial position feature; according to the weighted score result of the scoring vector, all grasp candidate areas are arranged in descending order according to the score value to generate a grasp priority ranking result.

7. The method for intelligent planning of a manipulator target grasping path based on visual depth perception according to claim 6 is characterized in that: The grasping posture parameters are composed of the grasping normal vector, the gripper rotation angle and the contact surface orientation, and are calculated based on the normal vector direction and neighborhood structure of the SuperPoint where the target grasping point is located.

8. The method for intelligent planning of a manipulator target grasping path based on visual depth perception according to claim 7 is characterized in that: The generation of the optimal grasping path trajectory data includes: A six-dimensional path state space is constructed based on the current position of the robot end and the three-dimensional coordinates of the target grasping point. The path search graph is constructed within the six-dimensional path state space by combining the obstacle boundary information in the SuperPoint graph structure. The inaccessible areas in the search graph are updated using the obstacle motion trend prediction results. The graph search algorithm is executed to generate a path trajectory sequence, which is then smoothed to generate the optimal grasping path trajectory data.

9. The method for intelligent planning of a manipulator target grasping path based on visual depth perception according to claim 8, characterized in that: The structure of the structural topology adaptive updating unit includes: The update module is used to collect new RGB images and depth images in real time during the execution of the path by the robotic arm to generate the current frame structured point cloud data set; the reconstruction module is used to recalculate the center of mass position, normal vector direction and adjacent edge set of the SuperPoint node based on the current frame structured point cloud data set; the matching module is used to match the node correspondence between the SuperPoint graph structure of the current frame and the graph structure of the previous frame, and identify node position drift and connection edge changes; the correction module is used to add, delete and adjust the weight of the connection edges in the SuperPoint graph structure according to the matching results to generate an updated SuperPoint graph structure.

10. The method for intelligent planning of a manipulator target grasping path based on visual depth perception according to claim 9, characterized in that: The generation process of the final grab control instruction includes the following strategy update mechanism: When the structural topology adaptive update unit detects that the center of mass position offset of the node where the grasping target is located in the SuperPoint graph structure in two consecutive frames of structured point cloud exceeds the set threshold, the grasping candidate area set is re-evaluated, the current target grasping point is replaced and the grasping posture parameters are updated; when any path segment in the optimal grasping path trajectory is judged to be an obstacle area in the corresponding position of the new frame of structured point cloud, the path search task is re-executed to generate new optimal grasping path trajectory data; when the action guidance graph response value has a score mutation in the grasping candidate area and the response value decreases by more than the set threshold, the improved SPGNet structure is called to recalculate the SuperPoint node features and re-arrange the grasping priority sorting results; when any strategy is triggered, the final grasping control instruction is recombined based on the updated target grasping point, grasping posture parameters and path trajectory data, and sent to the robot arm control end to execute the adjusted grasping action.

Images