Computer vision-based robotic arm grasping and positioning method

Abstract

The present application relates to the technical field of mechanical arm grabbing, and discloses a mechanical arm grabbing positioning method based on computer vision.The method comprises the following steps: acquiring a multi-view depth image sequence of a target object, and registering the multi-view depth image sequence to a unified coordinate system through a space-time alignment algorithm; completing three-dimensional reconstruction through an adaptive voxelization method, and generating a point cloud model of the target object; extracting surface curvature distribution features based on the point cloud model, dividing rigid and deformable regions, combining mechanical arm end effector parameters and motion constraints, and calculating a candidate grasp pose set in the rigid region; sorting and screening an optimal pose meeting a force closure condition through a dynamic weight evaluation function, and generating a collision-free grabbing path according to a kinematics inverse solution; and collecting force feedback data in real time during the grabbing process, and correcting a path planning sequence online.The method enhances the accuracy and adaptability of grabbing positioning, guarantees stable operation, and is suitable for various complex scenes.

Description

Technical Field

[0001] This invention relates to the field of robotic arm grasping technology, specifically a robotic arm grasping and positioning method based on computer vision. Background Technology

[0002] With the rapid development of industrial automation and artificial intelligence technologies, robotic arms, as the core execution units on automated production lines, directly impact production efficiency and product quality through their grasping and positioning accuracy and operational stability. They play an increasingly important role in numerous fields such as logistics sorting, automobile manufacturing, electronic assembly, and medical surgery. In the robotic arm's operational process, grasping and positioning is a crucial link connecting target recognition and execution. Its core requirement is to accurately perceive the state of the target object and plan a reasonable grasping strategy to ensure that the robotic arm's end effector can stably and efficiently complete the grasping action.

[0003] Currently, computer vision-based robotic arm grasping and positioning technology is widely used due to its non-contact perception advantages. However, in complex real-world scenarios, existing technologies still face many unresolved issues. Some methods rely on single-view image data for positioning analysis, which, limited by viewpoint occlusion and lack of depth information, makes it difficult to fully acquire the three-dimensional morphological features of the target object, easily leading to positioning errors. This is especially true for objects with irregular shapes or indistinct surface features, significantly increasing the risk of grasping failure. Even when some solutions introduce multi-view image acquisition mechanisms, the lack of effective spatiotemporal alignment strategies during image registration makes it difficult to accurately fuse image data from different viewpoints into a unified coordinate system, thus affecting the accuracy of subsequent 3D reconstruction.

[0004] In the 3D reconstruction process, traditional voxelization methods often use fixed voxel sizes. For complex target objects, if the voxel size is too large, detailed surface features will be lost; if the voxel size is too small, the data volume will surge, reducing reconstruction efficiency and the speed of subsequent feature extraction. Furthermore, existing technologies often ignore the differences in physical properties of different regions of the target object during grasping pose planning, failing to effectively distinguish between rigid and deformable regions. Randomly selecting grasping points on the object's surface may cause deformation or slippage due to grasping deformable regions, affecting grasping stability.

[0005] In the evaluation of candidate grasping poses, most methods use a single evaluation metric, failing to comprehensively consider factors such as robotic arm motion constraints and end effector geometric adaptability, making it difficult to select the optimal grasping pose. During the path planning stage, some solutions focus only on solving the inverse kinematics, neglecting the collision risks during robotic arm movement, which can easily lead to interference with the surrounding environment or other equipment. Furthermore, when the robotic arm performs grasping actions, dynamic changes in the external environment (such as slight object displacement or load fluctuations) are difficult to handle through pre-planned paths, lacking a real-time feedback and correction mechanism, further reducing the reliability and adaptability of grasping operations. These problems collectively limit the effectiveness of robotic arms in complex scenarios, necessitating a more comprehensive grasping and positioning method to address them. Summary of the Invention

[0006] The purpose of this invention is to provide a computer vision-based robotic arm grasping and positioning method to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides a robotic arm grasping and positioning method based on computer vision, the method comprising: A multi-view depth image sequence of the target object is acquired, and the multi-view depth image sequence is registered to a unified coordinate system using a spatiotemporal alignment algorithm; An adaptive voxelization method is used to perform 3D reconstruction on the registered depth image sequence to generate a point cloud model of the target object. The surface curvature distribution features of the target object are extracted based on the point cloud model, and rigid and deformable regions are divided according to the curvature distribution features. By combining the geometric parameters and motion constraints of the robotic arm's end effector, a set of candidate grasping poses is calculated within a rigid region. The candidate grasping poses are ranked for stability using a dynamic weight evaluation function, and the preferred grasping poses that satisfy the force closure condition are selected. Based on the inverse kinematics relationship between the preferred grasping pose and the current joint angle of the robotic arm, a collision-free grasping path planning sequence is generated. The system collects force feedback data in real time during the robotic arm's grasping process and corrects the grasping path planning sequence online based on the feedback data.

[0008] Preferably, the step of registering the multi-view depth image sequence to a unified coordinate system using a spatiotemporal alignment algorithm includes: Extract the edge feature point set from each frame of depth image, and construct a local reference coordinate system using the normal vector distribution of the feature point set; The transformation matrix of the local reference coordinate system between adjacent frames is calculated by the iterative nearest point algorithm, and the global registration parameters are accumulated. A nonlinear optimization method is used to eliminate the cumulative error in the global registration parameters, thereby minimizing the error in the overlapping area of ​​multi-view depth image sequences in a unified coordinate system.

[0009] Preferably, the step of performing 3D reconstruction of the registered depth image sequence using an adaptive voxelization method includes: Dynamically adjust the voxel mesh resolution according to the size range of the target object, and use a subdivision voxel strategy in high curvature regions; Gaussian filtering is applied to the point cloud data within each voxel unit to remove outliers, and then the voxel center coordinates are calculated. By connecting the center coordinates of adjacent voxels using the moving cube algorithm, a triangular mesh surface model with a topological structure is generated.

[0010] Preferably, the step of extracting the surface curvature distribution features of the target object based on the point cloud model includes: For each vertex in the triangular mesh surface model, construct a k-nearest neighbor search domain and calculate the eigenvalues ​​of the covariance matrix of the normal vectors within the search domain; The curvature type label of the vertex is determined based on the ratio of eigenvalues, and the region where the curvature type label changes continuously is marked as a deformable region; By merging adjacent vertices with the same curvature type label using a region growing algorithm, a closed boundary of a rigid region is formed.

[0011] Preferably, the step of calculating the candidate grasping pose set within the rigid region by combining the geometric parameters and motion constraints of the robotic arm end effector includes: Contact point pairs are uniformly sampled on the closed boundary of the rigid region, and the initial grasping pose is generated based on the gripping surface normal vector constraint of the end effector. Exclude the initial grasping pose that interferes with the robotic arm linkage model, and retain the effective grasping pose that meets the maximum gripping force requirement; The effective grasping pose is fine-tuned with six degrees of freedom to expand and form a set of candidate grasping poses that includes pose variation.

[0012] Preferably, the step of performing stability ranking on the candidate grasping pose set using a dynamic weight evaluation function includes: Calculate the force transfer matrix for each candidate grasping pose and evaluate the force closure index of the pose based on the matrix condition number. The friction cone constraint weights are dynamically adjusted based on the mass distribution characteristics of the target object to generate a stability scoring function. For each pose in the candidate grasp pose set, a stability scoring function is calculated, and the top N preferred grasp poses are selected after sorting them in descending order of the scores.

[0013] Preferably, generating a collision-free grasping path planning sequence includes: Discretize the inverse solution set of the optimal grasping pose in the joint space of the robotic arm and construct an reachability map; A bidirectional search algorithm is used to find the shortest path from the current pose to the target grasping pose in the reachability graph; By optimizing the joint angle change rate in the shortest path using the velocity potential field method, a smooth grasping path planning sequence is generated.

[0014] Preferably, the online correction of the crawling path planning sequence based on feedback data includes: The contact force vector output by the six-dimensional force sensor of the robotic wrist is monitored in real time, and the deviation from the expected contact force is calculated. When the deviation exceeds the threshold, the position compensation mechanism based on impedance control is triggered, generating a joint angle correction command. The joint angle correction command is inserted into the current execution node of the grasping path planning sequence to form a closed-loop control loop.

[0015] Preferably, the triggering of the impedance-controlled position compensation mechanism includes: Based on the stiffness characteristics of the target object, set the virtual spring damping parameters and establish a Cartesian space impedance model; The compensation displacement vector of the end effector is obtained by calculating the contact force deviation using the input impedance model. The compensation displacement vector is mapped to the torque adjustment amount in the joint space by using the Jacobian matrix transpose method.

[0016] Preferably, after forming the closed-loop control circuit, it further includes: Record the correspondence between the force feedback data and the compensation displacement vector for each online correction, and update the impedance model parameters; When the number of consecutive corrections exceeds the preset value, the stability sorting and path planning of the candidate grasping pose set are re-executed.

[0017] Compared with the prior art, the beneficial effects of the present invention are: By acquiring and spatiotemporally aligning multi-view depth image sequences, the limitations of single-view images in 3D information acquisition are effectively overcome. Multi-view data can capture the morphological features of target objects from different directions. Combined with spatiotemporal alignment algorithms, images from various perspectives can be accurately fused into a unified coordinate system, eliminating registration errors caused by perspective differences. This provides a more comprehensive and consistent data foundation for subsequent 3D reconstruction, enabling the reconstructed point cloud model to more realistically reflect the 3D structure of the target object and avoiding positioning deviations caused by missing information. This is particularly suitable for scenarios involving grasping target objects with irregular shapes or complex surface features.

[0018] The application of adaptive voxelization overcomes the bottleneck of traditional fixed voxelization methods. This method dynamically adjusts voxel sizes based on the structural complexity of different regions of the target object. Smaller voxels are used in areas with rich surface details to preserve accurate feature information, while larger voxels are used in areas with simple structures to reduce data redundancy, achieving a balance between reconstruction accuracy and efficiency. This dynamic adjustment mechanism not only improves the quality of 3D reconstruction but also reduces the amount of data in subsequent point cloud processing, accelerates feature extraction and pose planning, and makes the entire grasping and localization process more real-time, meeting the needs of efficient operations in industrial scenarios.

[0019] Dividing rigid and deformable regions based on surface curvature distribution characteristics provides an important basis for the rational selection of grasping poses. Objects of different materials and structures exhibit significant differences in surface curvature distribution. Curvature analysis can accurately identify rigid regions within an object that are not easily deformable, limiting the grasping point to these regions. This effectively avoids problems such as object damage and slippage caused by grasping deformable regions, fundamentally improving the stability of the grasping action. Simultaneously, this region division method makes the grasping strategy more targeted, eliminating the need for uniform analysis of the entire object and simplifying the complexity of pose planning.

[0020] During the calculation of candidate grasping poses, the feasibility of the candidate poses is ensured by combining the geometric parameters of the end effector and the motion constraints. The geometric parameters of the end effector directly determine its adaptability to the target object, while the motion constraints define the motion boundaries of the robotic arm within the workspace. Combining the two can eliminate pose options that do not meet the actual operational requirements, reduce invalid calculations, and make the generated set of candidate poses more closely match the actual working capabilities of the robotic arm, avoiding grasping failures caused by pose mismatch with the actuator or exceeding the motion range.

[0021] The introduction of a dynamic weighted evaluation function enables a comprehensive and objective assessment of candidate grasping poses. This function comprehensively considers factors such as force closure, pose stability, and actuator adaptability, and dynamically adjusts the weights of each factor according to the needs of different operational scenarios, breaking the limitations of traditional single-index evaluation. By ranking candidate poses based on their stability, the optimal pose that meets the force closure condition can be accurately selected, ensuring that the robotic arm can provide sufficient gripping force during grasping, preventing objects from loosening or falling off during handling, and further improving the reliability of grasping operations.

[0022] The collision-free grasping path planning sequence is generated based on inverse kinematics, balancing the rationality and safety of the path. Inverse kinematics can infer the motion angles of each joint of the robotic arm from the optimized grasping pose, ensuring that the path planning conforms to the kinematic characteristics of the robotic arm. At the same time, the surrounding environment and the robotic arm's own structure are fully considered during the path generation process, and the interference risk during the movement is avoided through a collision detection mechanism, ensuring that the robotic arm can complete the grasping action smoothly and safely in complex work spaces, avoiding equipment damage and production accidents.

[0023] Real-time force feedback and online correction mechanisms enable the robotic arm to dynamically adapt. During the grasping process, changes in the external environment are directly reflected in the force feedback data. By collecting and analyzing this data in real time, deviations between the planned path and the actual operation can be detected promptly, and the path sequence can be quickly corrected. This closed-loop control method breaks through the limitations of traditional open-loop planning, enabling the robotic arm to flexibly cope with uncertainties in dynamic scenarios, improving the adaptability and fault tolerance of grasping and positioning, and ensuring stable grasping performance even in complex and changing working environments. Attached Figure Description

[0024] Figure 1 This is a schematic diagram illustrating the working principle of the computer vision-based robotic arm grasping and positioning method described in this invention. Figure 2 A flowchart for registering multi-view depth image sequences to a unified coordinate system; Figure 3 The flowchart for adaptive voxelization 3D reconstruction. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Please see Figure 1This invention provides a computer vision-based robotic arm grasping and positioning method. The method includes: acquiring a multi-view depth image sequence of a target object, and registering the sequence to a unified coordinate system using a spatiotemporal alignment algorithm. This process ensures accurate fusion of depth data from different viewpoints and times. An adaptive voxelization method is used to perform 3D reconstruction on the registered depth image sequence, generating a point cloud model that accurately reflects the geometry of the target object. Based on the generated point cloud model, the surface curvature distribution features of the target object are further extracted, and the object surface is divided into rigid and deformable regions according to the curvature distribution features, providing a structural basis for subsequent grasping planning. Combining the specific geometric parameters and kinematic constraints of the robotic arm's end effector, a series of candidate grasping poses are calculated within the identified rigid regions. A dynamic weight evaluation function is used to perform stability analysis and ranking of the candidate grasping pose set, selecting the preferred grasping pose that satisfies the force closure condition. The inverse kinematics of the robotic arm is calculated based on the preferred grasping pose, and a collision-free motion trajectory sequence from the current position to the grasping point is generated. During the actual grasping process of the robotic arm, feedback data from the wrist force sensor is collected in real time, and the pre-generated grasping path is dynamically corrected online based on this feedback to adapt to changes in the actual contact environment and ensure the reliability and stability of the grasping.

[0027] Example 1: See Figure 2 In practice, the acquisition of multi-view depth image sequences is accomplished by using multiple depth cameras fixedly installed at different locations in the work scene. These cameras simultaneously acquire depth images of the target object under the triggering of a timestamp synchronization signal, thus forming a multi-view depth image sequence containing both spatial and temporal information. Extracting the edge feature point set from each frame of the depth image is the first step in the registration process. The edge feature point set is obtained by calculating the depth gradient of each pixel in the depth image. Pixels with gradient magnitudes exceeding a set threshold are identified as edge points. The set of three-dimensional coordinates of all these edge points constitutes the edge feature point set. A local reference coordinate system is constructed using the normal vector distribution of the edge feature point set. For each feature point in the edge feature point set, all point cloud data within a local neighborhood are selected centered on that point. Principal component analysis is performed on the point cloud within this neighborhood to calculate the eigenvalues ​​and eigenvectors of the covariance matrix. The eigenvector corresponding to the smallest eigenvalue is the normal vector direction at that point. Using the normal vector as an axis of the local reference coordinate system, and combining it with the tangent vector at that point, a complete local reference coordinate system capable of describing the local surface orientation is constructed.

[0028] In some embodiments, the transformation matrix of the local reference coordinate system between adjacent frames is calculated using the iterative nearest-point algorithm. The core of the iterative nearest-point algorithm is to find corresponding point pairs between two adjacent frame point cloud data. In a specific implementation, for each point in the current frame point cloud, the point with the closest Euclidean distance in the previous frame point cloud is found as the corresponding point, establishing a point pair relationship. Then, an objective function is constructed, which is the sum of the squares of the distances between all valid point pairs. The optimal rigid body transformation matrix is ​​solved by minimizing this objective function. This transformation matrix includes a rotation matrix and a translation vector, minimizing the overall distance between corresponding points in the two frame point clouds after transformation. The solution process typically uses singular value decomposition or quaternion methods to calculate the optimal rotation, and then derives the translation vector. The global registration parameters are obtained by performing chained multiplication on the transformation matrices between all adjacent frames. Assuming there is an image sequence from frame 1 to frame n, the transformation matrices from frame 1 to frame 2, from frame 2 to frame 3, and so on, up to the transformation matrices from frame (n-1) to frame n, are multiplied sequentially to obtain the overall transformation relationship from the coordinate system of frame 1 to the coordinate system of frame n. This allows all depth images to be initially transformed into a unified coordinate system based on the coordinate system of frame 1.

[0029] In practice, the Iterative Closest Point (TLP) algorithm introduces minute registration errors in each inter-frame registration step, which accumulate over time, leading to significant global registration drift and preventing the data from closing at the beginning and end of the sequence. To address this, a nonlinear optimization method is employed to eliminate these accumulated errors in the global registration parameters. This method typically uses a graph optimization framework, treating each depth image frame as a node in a graph and using the transformation matrices calculated by the TLP between adjacent frames as edges between nodes, thus constructing a pose graph. The objective function is the sum of constraint errors across all edges, specifically the difference between the predicted and actual measured values ​​of the transformation matrix. The entire graph is optimized using nonlinear least squares optimization algorithms such as the Levenberg-Marquardt algorithm or the Gauss-Newton algorithm, adjusting the pose estimates of all nodes to minimize the sum of squared errors across all constraint edges, effectively eliminating accumulated errors and achieving a globally consistent registration result. Minimizing the overlap error of multi-view depth image sequences in a unified coordinate system is the direct goal of optimization. During the optimization process, the point cloud data of the same object surface region observed from different viewpoints should overlap as much as possible. The optimization algorithm maximizes this overlap by adjusting the pose parameters of each frame, and finally achieves high-precision 3D alignment of the entire scene.

[0030] It is understandable that the stability of feature point extraction has a significant impact on the subsequent registration accuracy. In practical implementation, in addition to using depth gradients, the rate of change of the normal vector or curvature information of the point cloud can also be combined to comprehensively judge feature points, so as to extract a more representative and noise-robust feature point set. When constructing a local reference coordinate system, the choice of local neighborhood size needs to be balanced. If the neighborhood is too small, it will be sensitive to noise; if the neighborhood is too large, it will smooth out local details. Usually, the neighborhood radius or the number of nearest neighbors is dynamically adjusted according to the density of the point cloud and the noise level. Before executing the iterative nearest point algorithm, the point cloud can be downsampled to reduce the amount of data and improve the calculation speed, but it is necessary to ensure that the downsampled point cloud still retains the basic geometric features of the original point cloud. When constructing the optimization graph, not only can constraint edges between adjacent frames be added, but also closed-loop detection constraint edges between non-contiguous frames can be added. When the system detects that the current frame has an overlapping area with a historically distant frame, a closed-loop constraint edge is added. This can greatly enhance the global consistency of optimization and effectively correct cumulative drift. Nonlinear optimization processes involve a large amount of computation. In practical applications, incremental graph optimization methods can be used. Whenever new frame data is added, only local optimization is performed on the graph, rather than global re-optimization, in order to meet real-time requirements.

[0031] Example 2: See Figure 3In practical implementation, the process of 3D reconstruction of the registered depth image sequence using the adaptive voxelization method begins with the definition of spatial discretization. The core idea is to dynamically adjust the voxel mesh resolution according to the size range of the target object. Specifically, the bounding box of the target object's outer cube is obtained in advance, and the longest side of the bounding box is divided into a fixed number of intervals to calculate the initial voxel size. For larger objects, the initial voxel size is increased accordingly to control the overall data volume, while for smaller objects, a smaller initial voxel size is used to preserve details. In high-curvature regions, a subdivision voxel strategy is adopted. High-curvature regions are pre-identified by calculating the local curvature of the point cloud data before reconstruction. For regions where the curvature value exceeds a threshold, the initial large voxel covering the region is recursively subdivided in an octree manner, usually by one or two layers, thereby obtaining a higher resolution voxel mesh in that local area and achieving a fine expression of the sharp features and complex contours of the object's surface. Gaussian filtering is applied to the point cloud data within each voxel. The core of Gaussian filtering is to calculate the weighted average of the 3D coordinates of all points within the voxel. The weights are determined by a Gaussian function based on the distance from each point to the voxel center, thus smoothing out random noise. After outlier removal, the coordinates of the voxel center are calculated. Outlier removal is usually based on statistical methods, calculating the distribution of the average distance from all points within the voxel to their neighboring points. Points whose distance from the average distance exceeds three standard deviations are considered outliers and removed. Finally, the arithmetic mean or weighted center of the retained point cloud coordinates after filtering and denoising is determined as the final representative coordinates of the voxel. The moving cube algorithm connects the center coordinates of adjacent voxels. It traverses each voxel in the voxel mesh, checks the scalar values ​​at its eight vertices (these scalar values ​​can be derived from depth information or point cloud density), and determines the topological configuration of the isosurface within the voxel by comparing the magnitude of each vertex scalar value with a preset isosurface threshold. Based on a predefined lookup table, a series of triangular facets are generated inside the voxel. These triangular facets connect the isosurface information of adjacent voxels. Finally, all the triangular facets generated in the voxels are stitched together to form a triangular mesh surface model with a continuous topological structure. This model explicitly expresses the surface geometry of the target object.

[0032] In some embodiments, the surface curvature distribution features of the target object are extracted based on a triangular mesh surface model. A k-nearest neighbor search domain is constructed for each vertex in the triangular mesh surface model. The k-nearest neighbor search domain is the set of the k nearest vertices along the edge connections of the current vertex on the triangular mesh. The value of k affects the local characteristics of curvature calculation. The covariance matrix of the normal vectors within the k-nearest neighbor search domain is calculated. This covariance matrix is ​​obtained by summing the outer products of the differences between the normal vectors of all vertices in the search domain and their average normal vector. Eigenvalue decomposition is performed on the covariance matrix to obtain three eigenvalues. The magnitude of the eigenvalues ​​reflects the degree of change of the normal vector in the direction of the corresponding eigenvector. The curvature type label of the vertex is determined based on the ratio of the eigenvalues. A commonly used ratio is the ratio of the largest eigenvalue to the sum of the eigenvalues. The larger the ratio, the more concentrated the change in the normal vector direction, i.e., the greater the curvature. Based on a preset ratio threshold, vertices are classified into different curvature type labels such as flat points, edge points, or corner points.

[0033] It's understandable that marking regions with continuously changing curvature type labels as deformable regions is a judgment based on region consistency. Deformable regions typically correspond to soft, easily bent, or potentially elastically deformable parts of an object's surface. These regions often exhibit continuous curvature type label distribution and high values ​​on triangular meshes. The region growing algorithm merges adjacent vertices with the same curvature type label. The algorithm first selects a seed vertex, typically a point with representative curvature values. Starting from the seed vertex, it checks all its adjacent vertices. If an adjacent vertex has the same or similar curvature type label as the seed vertex, it merges that vertex into the current region. The newly merged vertex then recursively expands the region, using it as the new growth point, until no more adjacent vertices satisfying the conditions can be merged, ultimately forming connected regions. Forming the closed boundary of a rigid region is accomplished by identifying and connecting the edges of these homogeneous regions. Rigid regions are typically composed of vertices labeled with curvature type as flat points or low curvature. After region growth, each rigid region is a large set of connected vertices. The boundary of this vertex set consists of a series of edges connected end to end. One side of these edges belongs to the rigid region, and the other side belongs to other regions (such as high curvature regions or deformable regions). Extracting all these boundary edges and connecting them in sequence forms a closed polygonal boundary that surrounds the rigid region. This closed boundary clearly defines the potential contact surface that can be used for stable gripping.

[0034] Optionally, during the voxelization process, color information (if available) can be incorporated for finer partitioning, such as voxel subdivision at texture boundaries. When calculating vertex curvature, in addition to k-nearest neighbors, a fixed-radius neighborhood search can be used to adapt to uneven point cloud distributions. The similarity criterion in the region growing algorithm can be based not only on precise matching of curvature type labels but also on introducing continuous similarity of curvature values, allowing vertices with the same label but smooth curvature gradients to be merged to obtain larger, more complete rigid regions. Extraction of the closed boundaries of rigid regions can be achieved by efficiently traversing and tracing boundary edges using a half-edge data structure of the mesh. The entire 3D reconstruction and feature extraction process can utilize specialized geometry processing libraries to implement basic functions, such as voxelization filtering and moving cube surface reconstruction modules in point cloud libraries, and mesh operations and geometric query functions in computer graphics libraries. The adaptive volumetric approach effectively balances reconstruction accuracy and computational efficiency, while curvature-based region segmentation provides crucial semantic information about the object's structure for subsequent grasping planning. This involves dividing the object's surface into rigid parts suitable for gripping and deformable parts that need to be avoided or specially treated, which is a key step in achieving intelligent and reliable grasping.

[0035] Example 3: In specific implementation, the process of calculating the candidate grasping pose set within a rigid region by combining the geometric parameters and motion constraints of the robotic arm's end effector begins with utilizing the closed boundary of the rigid region. This closed boundary consists of a series of continuous three-dimensional points, which define the suitable grasping area contour on the object's surface. Contact point pairs are uniformly sampled along the closed boundary using equal arc lengths or equal angle intervals. Two points are selected each time as a potential contact point pair. The baseline distance between point pairs is limited by the maximum opening of the end effector, ensuring the gripper can physically reach these points. An initial grasping pose is generated based on the normal vector constraints of the end effector's gripping surface. For each pair of sampled contact points, the normal vector direction of the object's surface at each point is calculated. Then, the virtual gripper's posture is adjusted so that the normal vectors of the two gripping panels are aligned with the surface normal vectors at the corresponding contact points or form a predetermined safety angle. Simultaneously, the gripper's center point is ensured to be near the midpoint of the two contact points, thus generating a preliminary gripping posture that conforms to the contact geometry. To exclude initial grasping poses that interfere with the robotic arm's linkage model, a simplified 3D model of the robotic arm, the end effector model, and the environment model are loaded into the collision detection system. Inverse kinematics is then solved for each initial grasping pose. If an inverse solution exists, the entire path from the initial pose to the grasping pose is checked for collisions with itself or the environment, and any poses causing collisions are discarded. Valid grasping poses that meet the maximum gripping force requirement are retained. This requires estimating the force exerted by the gripper on the object in each grasping pose, based on the estimated weight of the object and the geometry of the gripping point. This ensures that the required gripping force is within the rated maximum gripping force range of the end effector, preventing slippage or overload. The effective grasping pose is fine-tuned in six degrees of freedom. This fine-tuning involves making small perturbations in three translational and three rotational directions, such as adding a small random offset in each direction or scanning with a fixed step size, to explore stability changes in the neighborhood of the grasping pose. This expands to form a set of candidate grasping poses containing pose variations, which covers more possible grasping configurations and increases the chance of finding robust grasping.

[0036] In some embodiments, the candidate grasping poses are ranked for stability using a dynamic weight evaluation function, and the force transfer matrix for each candidate grasping pose is calculated. The force transfer matrix is ​​a linear transformation matrix that maps the forces at each contact point of the gripper to the resultant force and resultant torque at the object's center of mass. The force closure index of the pose is evaluated based on the condition number of the force transfer matrix. The condition number reflects the numerical stability of matrix inversion; a smaller condition number indicates better force transfer performance and stronger resistance to external disturbances. The force closure index is typically defined as the reciprocal or negative logarithm of the condition number, with a larger value indicating more stable grasping. The friction cone constraint weights are dynamically adjusted based on the mass distribution characteristics of the target object. The mass distribution characteristics are estimated using the object's 3D model and density information. For objects whose center of mass deviates from the geometric center or have irregular mass distributions, the weight of the friction cone constraint is increased because such objects are more prone to rotation or sliding during grasping. The friction cone constraint ensures that the forces at the contact points are located inside the friction cone, satisfying Coulomb's law of friction. A stability scoring function is generated. This function is a weighted sum model that integrates the force closure index, friction cone constraint satisfaction, and other possible factors. Its mathematical expression is: Where: symbol The final stability score representing the grasping pose is a dimensionless scalar value; a higher value indicates a more stable grasp. (Symbol) The weighting coefficient representing the force closure index is a positive real number used to adjust the importance of force closure in the overall score. (Symbol) The result of the force closure index is a dimensionless scalar, typically derived from the condition number of the force transfer matrix. (Symbol) The weighting coefficient representing the friction cone constraint is a positive real number. Its magnitude is dynamically adjusted based on the non-uniformity of the target object's mass distribution; the more non-uniform the mass distribution, the larger the weighting value is typically set. (Symbol) The metric representing the satisfaction of the friction cone constraint is a scalar between 0 and 1, reflecting the degree to which the contact force is located inside the friction cone under the grasping pose. A value closer to 1 indicates better satisfaction. Weighting coefficient. and The sum is usually normalized to 1 to ensure consistency in scoring.

[0037] It is understandable that the selection of sampling density for uniform sampling of contact points requires a trade-off between computational complexity and the completeness of grasping opportunities. Overly dense sampling generates a large number of poses, increasing the computational burden, while overly sparse sampling may miss the optimal grasping point. The construction of the force transfer matrix depends on assumptions about the local surface geometry and friction coefficient at the contact point; therefore, its accuracy directly affects the reliability of the evaluation results. The dynamic adjustment mechanism of the friction cone constraint weights is key to improving the adaptability of the grasping strategy. For unknown objects, this can be achieved through online estimation or initialization using default parameters.

[0038] In some embodiments, the specific step size for six-DOF fine-tuning depends on the scene scale and is typically set to small amounts on the millimeter and degree levels to explore meaningful neighborhoods within a reasonable timeframe. The collision detection process can utilize acceleration structures such as hierarchical bounding box trees to improve computational efficiency and meet the real-time requirements of online planning. The weighting coefficients in the stability scoring function can be calibrated using experimental or simulation data, or designed to be adaptive and updated online based on historical capture success rates.

[0039] It is understandable that the generation and evaluation of candidate grasping poses is a multi-objective optimization process, requiring the search for satisfactory solutions under multiple constraints such as force closure, interference resistance, and collision-free operation. The ranking of the optimal grasping poses provides priority for subsequent path planning, allowing the planner to prioritize the poses with the highest scores, thereby improving the overall efficiency of the grasping system.

[0040] Optionally, when calculating the closure index, in addition to the condition number, other linear algebraic indices such as the rank of the grasping matrix or the minimum singular value can be considered to comprehensively evaluate the stability of the grasp. For very complex object shapes, the rigid region can be further divided into smaller sub-regions, and sampling and evaluation can be performed independently within different sub-regions to better utilize the graspable surface. When generating the candidate grasping pose set, not only can it be based on uniform sampling, but also learning-based methods can be introduced to predict potential high-quality grasping points, thereby guiding the sampling process and improving search efficiency.

[0041] Example 4: In specific implementation, generating a collision-free grasping path planning sequence begins with the discretization of the inverse kinematics of the robotic arm. This involves discretizing the set of inverse kinematics solutions for the preferred grasping pose within the robotic arm's joint space. For a six-DOF or higher robotic arm, the grasping pose of the same end effector may correspond to multiple different combinations of joint angles, i.e., multiple sets of inverse kinematics solutions. These continuous solution spaces need to be discretized. The discretization method is to set an angle resolution for each joint, for example, every 1 degree or every 0.5 degrees as an interval, thereby transforming the continuous joint angle space into a high-dimensional grid composed of a finite number of discrete joint angle points. The reachability map is constructed based on these discrete points. The reachability map is a graph data structure, where each node represents a specific discrete joint angle configuration of the robotic arm. If the poses represented by two nodes can be directly transformed through tiny movements of all joints (moving by an angular interval), and all intermediate poses traversed during this tiny movement pass collision detection, then an undirected edge is added between these two nodes to indicate that they are directly reachable. In this way, the discretized joint spaces are connected into a network representing the safe movement relationships of the robotic arm. A bidirectional search algorithm is used to find the shortest path from the current pose to the target grasping pose in the reachability graph. The bidirectional search algorithm searches in two directions simultaneously. One search process starts from the starting node representing the current joint angle of the robotic arm and expands outward to neighboring nodes; the other search process starts from the target node representing a certain inverse kinematic solution of the target grasping pose and also expands outward. The two search processes alternate until the boundaries of the two searches coincide, that is, a path connecting the starting and ending points has been found. This bidirectional search strategy is generally more efficient than unidirectional search. The rate of change of joint angles in the shortest path is optimized by the velocity potential field method. The shortest path obtained directly from the reachability graph may consist of a series of broken line segments in the joint space, resulting in unsmooth robotic arm movement and abrupt velocity changes. The velocity potential field method assigns a virtual potential energy value to each node on the path. The potential energy value prompts the node to move towards its neighboring node to smooth the path. At the same time, it constrains the change of joint angle between adjacent nodes to not exceed the maximum allowable speed and acceleration of the motor and reducer. By iteratively adjusting the position of the path nodes, a grasping path planning sequence with continuous joint angle changes, smooth motion and dynamic constraints is finally generated.

[0042] Referring to Table 1, real-time acquisition of force feedback data during the robotic arm's grasping process and online correction of the grasping path planning sequence based on this feedback data are key to achieving adaptive grasping. Real-time monitoring of the contact force vector output by the six-dimensional force sensor on the robotic wrist is crucial. The six-dimensional force sensor can simultaneously measure forces in three directions. , , ) and torque in three directions ( , , These data are read from sensors at a fixed high frequency (e.g., 1000 Hz). The deviation between the contact force vector and the expected contact force is calculated. The expected contact force is a theoretical value estimated based on the object's weight, center of mass, and grasping posture during the grasping planning phase. The actual measured six-dimensional force vector is subtracted element-wise from the expected value to obtain the force and torque deviations in the six dimensions. When the deviation exceeds a threshold, an impedance-controlled position compensation mechanism is triggered. The threshold is a safety range set individually for each force / torque component. For example, the deviation threshold for the normal contact force might be set to ±5 Newtons, and the torque threshold to ±0.1 Nm. Compensation is triggered when the absolute value of the deviation of any component exceeds its threshold. Generating joint angle correction commands is the core of the compensation mechanism. The impedance control model calculates the small displacement adjustment (including translation and rotation) required by the end effector based on the force deviation. This adjustment is converted into a correction for the current desired trajectory point, and then the angle correction amount to be applied to each joint is solved through the inverse kinematics of the robotic arm. When a joint angle correction command is inserted into the current execution node of the grasping path planning sequence, the robotic arm's control system does not interrupt the current path execution. Instead, it adds the calculated joint angle correction as an increment to the joint angle command being sent to the servo driver in real time. This allows for fine-tuning of the robotic arm's trajectory without disrupting motion continuity, enabling it to adapt to actual contact conditions, such as minor unevenness on an object's surface or slight slippage. Forming a closed-loop control circuit means that this perception (force sensor) - decision (impedance control) - execution (joint correction) process runs cyclically throughout the grasping contact phase, continuously using the latest force feedback to correct the motion until the grasping action is successfully completed and the force deviation stabilizes within a threshold.

[0043] Table 1: Threshold for Deviation between Six-Dimensional Force Sensor Feedback Data and Expected Values

[0044] In some embodiments, the construction of the reachability map can be performed offline to save online computation time, especially for highly structured fixed work scenarios, where a dense reachability map covering commonly used workspaces can be pre-constructed. During online path planning, only the local subgraph containing the current attitude node and the target attitude node needs to be searched, significantly improving real-time performance. The parameters in the impedance control model, namely the virtual stiffness, damping, and mass coefficient, need to be carefully adjusted according to the specific dynamic characteristics of the robotic arm and the characteristics of the grasped object. For rigid objects and precision operations, higher stiffness and damping are usually used to obtain a fast force response, while lower stiffness is used for fragile or soft objects to avoid excessive contact force. The bidirectional search algorithm may face the combinatorial explosion problem when encountering high-dimensional joint spaces, so the discretization resolution needs to be carefully selected to achieve a balance between path quality and computational overhead. The real-time requirements for force feedback are very high. The control cycle of the entire closed loop, from data acquisition, deviation calculation, impedance model solution to command issuance, needs to be short enough (usually less than 10 milliseconds) to effectively suppress fluctuations in contact force.

[0045] Optionally, to further improve the smoothness and optimality of the path, after optimization using the velocity potential field method, spline curves can be used to interpolate discrete path points to generate a continuously differentiable joint space trajectory. For handling force deviations, dead-zone control can be employed; that is, when the deviation is within a small dead zone within a threshold range, no compensation is performed to avoid frequent system jitter due to minor noise. Online correction can not only correct the position but also trigger safety strategies when abnormally large force deviations are detected, such as immediately stopping the movement or slightly retracting the gripper to prevent damage to the robotic arm or object. The stability of the closed-loop control circuit is crucial; it is necessary to ensure the numerical stability and real-time performance of the impedance control algorithm to avoid system oscillations caused by computational delays or numerical errors.

[0046] Example 5: For a rigid object such as a metal part, the stiffness coefficient of the virtual spring can be set relatively high, for example, 5000 N / m, and the virtual damping coefficient can be set to 300 N·s / m to quickly respond and suppress contact force oscillations. For a soft object such as a sponge block, the stiffness coefficient of the virtual spring needs to be set relatively low, for example, 500 N / m, and the virtual damping coefficient can be set to 50 N·s / m to avoid excessive reaction force causing deformation of the object or overshoot of the robotic arm. A Cartesian space impedance model is established, which defines the differential equation of motion of the end effector in Cartesian space. Its general form relates the acceleration, velocity, position deviation, and contact force deviation of the end effector, and describes the dynamic behavior of the end effector through virtual mass, damping, and spring parameters. The contact force deviation is input into the impedance model to calculate the compensation displacement vector of the end effector. The contact force deviation is a six-dimensional vector containing deviations in three forces and three torques. The impedance model solves a second-order differential equation in real time based on this deviation vector, outputting the small displacement and rotational adjustment required by the end effector to compensate for the force deviation, i.e., a six-degree-of-freedom compensation displacement vector. The compensation displacement vector is mapped to the torque adjustment in joint space using the Jacobian matrix transpose method. The Jacobian matrix describes the linear transformation relationship between the Cartesian space velocity of the robotic arm end effector and the joint space velocity. The transpose of the Jacobian matrix can map the force in Cartesian space to the torque in joint space. The calculated compensation displacement vector is regarded as a generalized "virtual displacement". By left-multiplying the Jacobian matrix by a gain matrix, the compensation torque commands to be added to each joint can be calculated. These torque commands are sent to the joint servo driver to achieve real-time fine-tuning of the trajectory.

[0047] After forming a closed-loop control circuit, the system enters a continuous learning and monitoring state, recording the correspondence between force feedback data and compensation displacement vectors for each online correction. Each time the compensation mechanism is triggered, the system records the timestamp, the original six-dimensional contact force deviation vector, and the six-dimensional compensation displacement vector calculated through the impedance model as a data pair in a circular buffer or log file. Updating the impedance model parameters is based on recorded historical data. Parameter identification techniques such as recursive least squares can be used, using the recorded force deviation and compensation displacement data as input-output samples to estimate virtual spring, damping, and mass parameters that better reflect the current dynamic interaction between the robotic arm and the object. The updated parameters then replace the old parameters in the impedance model, enabling the model to adapt to different objects and contact conditions. When the number of consecutive corrections exceeds a preset value, the stability ranking and path planning of the candidate grasping pose set are re-executed. The preset value is an empirical threshold; for example, if force compensation is triggered 10 times consecutively within 2 seconds, it indicates that the current grasping pose has encountered continuous and significant interference during actual execution, and the pose itself may no longer be the optimal choice in the changed environment. At this point, the control system sends an interrupt signal to pause the current grasping action, instructs the robotic arm to safely retreat from the object, and then restarts the process that started from 3D perception, or at least re-calls the previously calculated set of candidate grasping poses, re-evaluates and sorts the stability based on the latest force interaction experience (for example, poses that frequently trigger compensation can be considered to have poor stability), and re-plans the motion path based on the new preferred pose. This is a task-level replanning to ensure long-term success.

[0048] In some embodiments, the initial setting of virtual spring damping parameters can be based on an object material library, which pre-stores recommended impedance parameters for common materials. The operator only needs to specify the object material, and the system can automatically load the corresponding initial parameters. When updating impedance model parameters, the recursive least squares method typically introduces a forgetting factor, giving new data higher weight, thus enabling the model to quickly track changes in interactive dynamics. The Jacobian matrix changes in real time with the configuration of the robotic arm; therefore, when calculating joint torque using the Jacobian matrix transpose method, the Jacobian matrix under the current posture of the robotic arm needs to be calculated in real time. Recording the correspondence between force feedback data and compensation displacement vectors is not only used for parameter updates but also for offline analysis and debugging of the grasping process, helping engineers understand the details of force interaction during grasping. The threshold setting for the number of consecutive corrections needs to be carefully considered; a threshold that is too low may lead to excessively frequent replanning, affecting efficiency, while a threshold that is too high may cause the system to waste too much time on suboptimal grasping.

[0049] Optionally, a conservative strategy can be adopted when updating impedance model parameters, i.e., only making small adjustments to the parameters each time to avoid system instability caused by drastic changes in model parameters due to a single abnormal data point. Besides complete reprogramming, when the number of consecutive corrections exceeds a limit, an alternative simplification strategy can be tried, such as directly selecting the pose with the second-best stability score from the candidate grasping pose set without re-performing 3D perception. When recording data, not only force deviations and compensation displacements can be recorded, but also information such as the joint angles of the robotic arm and the pose of the end effector can be recorded simultaneously, providing a more comprehensive context for analysis. For critical grasping tasks, a two-level warning system can be set: a warning is issued when the number of consecutive corrections reaches a lower threshold without interrupting the task; reprogramming is only performed when a higher threshold is exceeded. The entire impedance-based online correction and learning loop constitutes a typical adaptive control system, capable of adjusting its behavior through online interaction in uncertain environments. It should be noted that in this paper, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0050] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A robotic arm grasping and positioning method based on computer vision, characterized in that, include: A multi-view depth image sequence of the target object is acquired, and the multi-view depth image sequence is registered to a unified coordinate system using a spatiotemporal alignment algorithm; An adaptive voxelization method is used to perform 3D reconstruction on the registered depth image sequence to generate a point cloud model of the target object. The surface curvature distribution features of the target object are extracted based on the point cloud model, and rigid and deformable regions are divided according to the curvature distribution features. By combining the geometric parameters and motion constraints of the robotic arm's end effector, a set of candidate grasping poses is calculated within a rigid region. The candidate grasping poses are ranked for stability using a dynamic weight evaluation function, and the preferred grasping poses that satisfy the force closure condition are selected. Based on the inverse kinematics relationship between the preferred grasping pose and the current joint angle of the robotic arm, a collision-free grasping path planning sequence is generated. The system collects force feedback data in real time during the robotic arm's grasping process and corrects the grasping path planning sequence online based on the feedback data.

2. The robotic arm grasping and positioning method based on computer vision according to claim 1, characterized in that, The step of registering the multi-view depth image sequence to a unified coordinate system using a spatiotemporal alignment algorithm includes: Extract the edge feature point set from each frame of depth image, and construct a local reference coordinate system using the normal vector distribution of the feature point set; The transformation matrix of the local reference coordinate system between adjacent frames is calculated by the iterative nearest point algorithm, and the global registration parameters are accumulated. A nonlinear optimization method is used to eliminate the cumulative error in the global registration parameters, thereby minimizing the error in the overlapping area of ​​multi-view depth image sequences in a unified coordinate system.

3. The robotic arm grasping and positioning method based on computer vision according to claim 2, characterized in that, The method of using adaptive voxelization to perform 3D reconstruction of the registered depth image sequence includes: Dynamically adjust the voxel mesh resolution according to the size range of the target object, and use a subdivision voxel strategy in high curvature regions; Gaussian filtering is applied to the point cloud data within each voxel unit to remove outliers, and then the voxel center coordinates are calculated. By connecting the center coordinates of adjacent voxels using the moving cube algorithm, a triangular mesh surface model with a topological structure is generated.

4. The robotic arm grasping and positioning method based on computer vision according to claim 3, characterized in that, The extraction of surface curvature distribution features of the target object based on the point cloud model includes: For each vertex in the triangular mesh surface model, construct a k-nearest neighbor search domain and calculate the eigenvalues ​​of the covariance matrix of the normal vectors within the search domain; The curvature type label of the vertex is determined based on the ratio of eigenvalues, and the region where the curvature type label changes continuously is marked as a deformable region; By merging adjacent vertices with the same curvature type label using a region growing algorithm, a closed boundary of a rigid region is formed.

5. The robotic arm grasping and positioning method based on computer vision according to claim 4, characterized in that, The calculation of a set of candidate grasping poses within a rigid region, combining the geometric parameters and motion constraints of the robotic arm's end effector, includes: Contact point pairs are uniformly sampled on the closed boundary of the rigid region, and the initial grasping pose is generated based on the gripping surface normal vector constraint of the end effector. Exclude the initial grasping pose that interferes with the robotic arm linkage model, and retain the effective grasping pose that meets the maximum gripping force requirement; The effective grasping pose is fine-tuned with six degrees of freedom to expand and form a set of candidate grasping poses that includes pose variation.

6. The robotic arm grasping and positioning method based on computer vision according to claim 5, characterized in that, The stability ranking of the candidate grasping pose set using a dynamic weight evaluation function includes: Calculate the force transfer matrix for each candidate grasping pose and evaluate the force closure index of the pose based on the matrix condition number. The friction cone constraint weights are dynamically adjusted based on the mass distribution characteristics of the target object to generate a stability scoring function. For each pose in the candidate grasp pose set, a stability scoring function is calculated, and the top N preferred grasp poses are selected after sorting them in descending order of the scores.

7. The robotic arm grasping and positioning method based on computer vision according to claim 6, characterized in that, The generation of a collision-free crawling path planning sequence includes: Discretize the inverse solution set of the optimal grasping pose in the joint space of the robotic arm and construct an reachability map; A bidirectional search algorithm is used to find the shortest path from the current pose to the target grasping pose in the reachability graph; By optimizing the joint angle change rate in the shortest path using the velocity potential field method, a smooth grasping path planning sequence is generated.

8. The robotic arm grasping and positioning method based on computer vision according to claim 7, characterized in that, The online correction of the crawling path planning sequence based on feedback data includes: The contact force vector output by the six-dimensional force sensor of the robotic wrist is monitored in real time, and the deviation from the expected contact force is calculated. When the deviation exceeds the threshold, the position compensation mechanism based on impedance control is triggered, generating a joint angle correction command. The joint angle correction command is inserted into the current execution node of the grasping path planning sequence to form a closed-loop control loop.

9. The robotic arm grasping and positioning method based on computer vision according to claim 8, characterized in that, The triggering-based impedance-controlled position compensation mechanism includes: Based on the stiffness characteristics of the target object, set the virtual spring damping parameters and establish a Cartesian space impedance model; The compensation displacement vector of the end effector is obtained by calculating the contact force deviation using the input impedance model. The compensation displacement vector is mapped to the torque adjustment amount in the joint space by using the Jacobian matrix transpose method.

10. The robotic arm grasping and positioning method based on computer vision according to claim 9, characterized in that, After forming the closed-loop control circuit, it also includes: Record the correspondence between the force feedback data and the compensation displacement vector for each online correction, and update the impedance model parameters; When the number of consecutive corrections exceeds the preset value, the stability sorting and path planning of the candidate grasping pose set are re-executed.

Images