Visual identification grabbing method based on multi-degree-of-freedom mechanical arm

Abstract

The invention relates to the technical field of mechanical arm control, and discloses a multi-degree-of-freedom mechanical arm-based visual identification grabbing method, which comprises the following steps of: acquiring point cloud data of a target object by utilizing a depth camera, and screening the point cloud data to determine a target feature point; the target feature points are converted into mechanical arm base coordinate system pose data; solving an optimal joint angle; acquiring tail end pose data fed back by the depth camera in real time, and calculating a joint angle correction amount based on a preset formula and the tail end pose data; a control signal is output through the PID controller, and the seven-degree-of-freedom carbon fiber mechanical arm is controlled to grab a target object; and whether the seven-degree-of-freedom carbon fiber mechanical arm successfully grabs the target object or not is verified through point cloud collision detection. By implementing the method, the problems that mechanical arm visual grabbing has obvious defects in the aspects of flexibility of degree of freedom, visual perception robustness, control cooperation precision, grabbing verification reliability and the like, and high-precision and high-stability grabbing requirements in a complex environment are difficult to adapt are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mechanical arm control, and particularly relates to a visual recognition and grabbing method based on a multi-degree-of-freedom mechanical arm. BACKGROUND

[0002] With the rapid development of industrial automation and intelligent manufacturing, the mechanical arm, as the core equipment of flexible production, has evolved from traditional fixed trajectory operation to integrated intelligent grabbing. Among them, the introduction of visual guidance technology is the key to promoting this change. Through the acquisition of three-dimensional information of the target object by a depth camera, the mechanical arm can break through the limitations of the preset trajectory and adapt to the dynamically changing grabbing scene.

[0003] Traditional mechanical arms are mostly designed with four or six degrees of freedom, and their kinematic models are simple and their inverse solutions are unique. However, the lack of redundant degrees of freedom leads to insufficient obstacle avoidance capability and posture flexibility. In a cluttered environment, if there are obstacles around the target object, such as supports beside the assembly line or stacked parts, such mechanical arms often have to give up grabbing because they cannot adjust the posture of the intermediate joints. At the same time, the connecting rods of such mechanical arms are mostly made of metal, and the high motion inertia not only limits the dynamic response speed, but also exacerbates joint wear and reduces the service life of the equipment. In terms of visual recognition, most rely on the target features of "specific color + regular shape", and achieve positioning through simple color threshold segmentation and edge detection. However, when facing transparent objects, shiny surfaces or irregular shapes, the recognition accuracy decreases significantly, and even the target is lost. This defect is particularly prominent in scenes involving diversified materials such as consumer electronics and food processing. In addition, most systems use an open-loop control mode of "one-time coordinate transformation", that is, visual data is only used for initial positioning, and the end pose drift caused by connecting rod deformation and load changes during the motion of the mechanical arm is not considered. The final grabbing error often exceeds ±5mm, which is difficult to meet the high-precision requirements of precision assembly and other high-precision requirements.

[0004] In addition, the conversion of the coordinate system of the ordinary depth camera and the base coordinate system of the mechanical arm mostly relies on static calibration. However, in long-term operation, the slight loosening of the camera support and the vibration of the mechanical arm base will cause the calibration parameters to drift, causing the coordinate transformation error to accumulate. At the same time, the motion planning algorithm fails to fully utilize the real-time nature of visual feedback. The inverse kinematics solution of the four-degree-of-freedom mechanical arm only focuses on whether the end pose meets the requirements, while ignoring the optimization space that can be excavated by the redundant degrees of freedom. Even if some six-degree-of-freedom mechanical arms introduce redundant control, their optimization targets are mostly limited to a single indicator, and a multi-objective balance of "pose accuracy-energy consumption-joint protection" has not been formed. In actual application, this will cause the mechanical arm to excessively rotate the joints to avoid obstacles when grabbing, which not only increases energy consumption, but also may cause the joints to be in extreme positions for a long time, shortening the service life.

[0005] In summary, the related mechanical arm visual grasping technology has significant deficiencies in flexibility of degrees of freedom, robustness of visual perception, control coordination accuracy, and reliability of grasping verification, and is difficult to adapt to high-precision and high-stability grasping requirements in complex environments. SUMMARY

[0006] Therefore, the application provides a multi-degree-of-freedom mechanical arm visual recognition grasping method to solve the problem that the related mechanical arm visual grasping technology has significant deficiencies in flexibility of degrees of freedom, robustness of visual perception, control coordination accuracy, and reliability of grasping verification, and is difficult to adapt to high-precision and high-stability grasping requirements in complex environments.

[0007] The application provides a multi-degree-of-freedom mechanical arm visual recognition grasping method, which is applied to a multi-degree-of-freedom mechanical arm, and the multi-degree-of-freedom mechanical arm comprises a seven-degree-of-freedom carbon fiber mechanical arm, a depth camera, and a controller. The depth camera is installed on a base of the seven-degree-of-freedom carbon fiber mechanical arm, and the method comprises the following steps: acquiring point cloud data of a target object by using the depth camera, screening the point cloud data to determine target feature points, converting the target feature points into base coordinate system pose data of the mechanical arm, solving optimal joint angles based on the coordinate system pose data, a preset multi-target optimization model, and an obstacle avoidance constraint condition, acquiring end pose data fed back by the depth camera in real time, calculating a joint angle correction amount based on a preset formula and the end pose data, outputting a control signal by using a PID controller based on the optimal joint angles and the joint angle correction amount, controlling the seven-degree-of-freedom carbon fiber mechanical arm to grasp the target object, and verifying whether the seven-degree-of-freedom carbon fiber mechanical arm successfully grasps the target object by point cloud collision detection.

[0008] The embodiment provides a multi-degree-of-freedom mechanical arm vision recognition and grabbing method, first, point cloud data of a target object is acquired by using a depth camera, target feature points are determined by screening the point cloud data, the semantic type of each point is accurately judged by combining a semantic segmentation model, and through preset confidence threshold and mechanical arm workspace depth range constraint, the effective information truly related to the grabbing task is stripped from the mass point cloud of the complex scene. Breakthrough the limitation of traditional visual recognition which only relies on color or simple shape features, even if facing transparent objects, shiny surfaces or irregular shaped target objects, the target can also be locked through the spatial distribution and reflection intensity characteristics of three-dimensional point cloud. Secondly, the target feature points are converted into mechanical arm base coordinate system pose data, realizing accurate mapping from visual perception space to mechanical arm motion space. The conversion matrix obtained based on the calibration board strictly maps the camera coordinate system coordinates output by the depth camera to the mechanical arm base coordinate system, eliminating the spatial deviation between different sensors. Through solving the optimal joint angle based on the coordinate system pose data, the preset multi-target optimization model and the obstacle avoidance constraint condition, the redundancy characteristics of the seven-degree-of-freedom mechanical arm are fully utilized. The multi-target optimization model does not pursue single end pose accuracy, but considers pose accuracy, motion energy consumption and joint limit protection: through the weight coefficient, the priority of different targets is balanced, while ensuring that the end can accurately reach the target position, the intermediate joint attitude is adjusted to avoid obstacles by using the redundant freedom, and the joint motion energy consumption is minimized, and the joint is prevented from being in the limit position for a long time to avoid aggravating wear. Compared with the simple trajectory generation of traditional few-degree-of-freedom mechanical arms, it can adapt to more complex environments and prolong the service life of the mechanical arm, thereby realizing the coordinated optimization of flexibility, safety and economy. Through acquiring the end pose data fed back by the depth camera in real time, the joint angle correction amount is calculated based on a preset formula, and a dynamic closed-loop control mechanism is constructed. In the process of mechanical arm movement, the actual pose of the end is continuously monitored by using the depth camera, the deviation matrix is generated by comparing the theoretical planning pose, and the pose deviation is converted into the joint angle correction amount through the Jacobian matrix. Real-time correction can dynamically compensate the pose drift of the mechanical arm caused by link deformation, load change or environmental interference, so as to ensure that the end effector can accurately track the target position, and the problem of error accumulation in traditional open-loop control is avoided. Then, the PID controller outputs the control signal based on the optimal joint angle and the joint angle correction amount, realizing the smoothness and stability of the mechanical arm movement. Through the synergistic effect of proportional, integral and differential coefficients, the PID control quickly responds to the joint angle correction demand: the proportional term adjusts the deviation immediately, the integral term eliminates the accumulated error, and the differential term suppresses the action overshoot. It ensures that the mechanical arm moves smoothly when adjusting the attitude or approaching the target object, avoids the target object from sliding or the mechanical arm from vibrating caused by sudden movement, and improves the reliability of the grabbing execution.The success of grabbing the target object is verified through point cloud collision detection, objective and quantitative grabbing state judgment criteria are established, the point clouds of the end effector and the target object are segmented, the minimum contact distance and the contact area ratio of the two are calculated, and it is accurately judged whether a stable clamping is formed from the perspective of three-dimensional space. When the contact distance is less than the set threshold and the contact area ratio meets the requirements, it can be determined that the grabbing is successful, otherwise the retry mechanism is triggered. The "virtual cover" and "actual clamping" are effectively distinguished, the misjudgment rate is reduced, and the overall grabbing success rate is further improved through the closed-loop verification and retry mechanism, thereby reducing the need for manual intervention. Through the implementation of the application, the problems that the related mechanical arm visual grabbing technology has significant deficiencies in flexibility, visual perception robustness, control coordination accuracy and grabbing verification reliability, and is difficult to adapt to high-precision and high-stability grabbing requirements in complex environments are solved.

[0009] In an optional implementation, point cloud data of the target object is acquired by using a depth camera, and target feature points are determined by screening the point cloud data, including: The depth camera collects point cloud data P of the target object, , is the i-th point, is a three-dimensional coordinate, is a reflection intensity, the point cloud data is input into a semantic segmentation model to obtain a semantic probability of each point The semantic type corresponding to the semantic probability includes: target object type, background type and obstacle type. The target feature points are determined by screening the point cloud data based on a screening formula, and the screening formula is as follows: (1) wherein, represents the target feature point, represents a confidence threshold, and the value range is [0.6, 0.8], represents the depth range of the working space of the seven-degree-of-freedom carbon fiber mechanical arm.

[0010] In an optional implementation, the target feature points are converted into mechanical arm base coordinate system pose data, and the conversion formula is as follows: (2) wherein, represents the coordinates of the target feature point in the camera coordinate system, which is directly output by the depth camera, represents the coordinates of the converted target feature point in the mechanical arm base coordinate system, represents a conversion matrix, which is obtained by calibration of a calibration plate and contains a rotation matrix R and a translation vector t.

[0011] In an optional implementation, in a preset multi-objective optimization model, The optimization objective function is as follows:

[0012] wherein the weight coefficient ; The sub-objective is defined as follows: pose accuracy wherein is the expected end pose matrix, is the actual pose matrix, is the F-norm; energy consumption wherein is the joint energy consumption coefficient of the kth joint, which is positively correlated with the joint inertia, is the joint angular velocity of the kth joint; joint limit penalty wherein is the safety threshold value, is the maximum limit angle of the kth joint, is the attenuation factor, which triggers the penalty when .

[0013] In an optional embodiment, the obstacle avoidance constraint condition is as follows: (4) wherein is the shortest distance between the link of the seven-degree-of-freedom carbon fiber robot arm and the obstacle, is the safety distance.

[0014] In an optional embodiment, based on the coordinate system pose data, the preset multi-objective optimization model and the obstacle avoidance constraint condition, the optimal joint angle is solved, comprising: the optimal joint angle is solved by using the gradient descent method for iteration , and the formula is as follows: (5) wherein is the learning rate, is the gradient of the optimization objective function at , and the iteration is converged to .

[0015] In an optional embodiment, the end pose data fed back by the depth camera in real time is acquired, and the joint angle correction amount is calculated based on the preset formula and the end pose data, comprising: the end pose data fed back by the depth camera in real time is acquired, and it is assumed that the actual end pose detected by the depth camera at is , is a 4x4 matrix, the theoretical pose of inverse kinematics planning is The deviation matrix is established as follows: (6) wherein, is the inverse matrix of the theoretical pose; The mapping relationship between joint angular velocity and end velocity is described by the Jacobian matrix of the mechanical arm The pose deviation is converted into joint angle correction, and the conversion formula is as follows: (7) wherein, is a 6x7 matrix, is the pseudo-inverse of the Jacobian, is a regularization parameter for avoiding matrix singularity, is the vectorization operation of the deviation matrix, which is used to convert the 4x4 matrix into a 16x1 vector, and the first 6 elements correspond to the translation and rotation deviation.

[0016] In an optional embodiment, based on the optimal joint angle and the joint angle correction, a control signal is output by a PID controller to control the seven-degree-of-freedom carbon fiber mechanical arm to grasp the target object, comprising: The corrected joint angle is The control signal output by the PID controller is as follows: (8) wherein, is a proportional coefficient, is an integral coefficient, is a differential coefficient, , The integral operation is performed on the joint angle deviation.

[0017] In an optional embodiment, whether the target object is successfully grasped is verified by point cloud collision detection, comprising: After the grasping action is completed, the point cloud data of the current scene is collected ; wherein, P grasp is a set of point cloud data of the current scene collected by the depth camera, is the index of each point in the point cloud data, is the three-dimensional coordinates of the mth point in the camera coordinate system is the reflection intensity of the mth point; The point cloud data is denoised and ROI cropped, and the end effector point cloud and the target object point cloud are segmented; The end effector point cloud Point cloud of target object Minimum contact distance The formula is as follows: (9) in, for any point in, for any point in, This is the function for calculating Euclidean distance; Computational end effector point cloud Point cloud of target object Contact area ratio The formula is as follows: (10) in, The contact detection threshold; like and If the data is successfully scraped, it is considered scraped successfully; otherwise, it is considered scraped unsuccessfully.

[0018] In one alternative implementation, the method further includes: if the grasping is determined to have failed, triggering a retry mechanism to replan the grasping point and the end effector posture. Attached Figure Description

[0019] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a vision recognition and grasping method based on a multi-degree-of-freedom robotic arm according to an embodiment of the present invention. Detailed Implementation

[0021] With the rapid development of industrial automation and intelligent manufacturing, robotic arms, as core equipment in flexible production, have evolved from traditional fixed-track operations to integrated intelligent grasping. The introduction of vision-guided technology is key to this transformation. By acquiring 3D information about the target object through devices such as depth cameras, robotic arms can break through the limitations of preset trajectories and adapt to dynamically changing grasping scenarios.

[0022] Traditional manipulators mostly adopt four or six degrees of freedom design, whose kinematic model is simple and inverse solution is unique, but the lack of redundant freedom leads to insufficient obstacle avoidance ability and posture flexibility. In a cluttered environment, if there are obstacles around the target object, such as supports beside the assembly line and stacked parts, such manipulators often have to give up grabbing because they cannot adjust the posture of the intermediate joints; meanwhile, the metal material used in the connecting rods limits the dynamic response speed and exacerbates joint wear, reducing the service life of the equipment. Visual recognition mostly relies on the target features of "specific color + regular shape", and positioning is achieved through simple color threshold segmentation and edge detection, but when facing transparent objects, shiny surfaces or irregular shapes, the recognition accuracy drops significantly, and even the target is lost, which is particularly prominent in scenes involving diversified materials such as consumer electronics and food processing. In addition, most systems use an open-loop control mode of "one-time coordinate transformation", that is, visual data is only used for initial positioning, and the end pose drift caused by connecting rod deformation and load changes during the movement of the manipulator is not considered, so the final grabbing error often exceeds ±5mm, which is difficult to meet the high-precision requirements of precision assembly and other high-precision requirements.

[0023] In addition, the conversion of the coordinate system of the ordinary depth camera and the base coordinate system of the manipulator mostly relies on static calibration, but in long-term operation, the slight loosening of the camera support and the vibration of the manipulator base will cause the calibration parameters to drift, causing the coordinate transformation error to accumulate. At the same time, the motion planning algorithm fails to fully utilize the real-time nature of visual feedback, and the inverse kinematics solution of the four-degree-of-freedom manipulator only focuses on whether the end pose meets the requirements, while ignoring the optimization space that can be excavated by the redundant freedom. Even if some six-degree-of-freedom manipulators introduce redundant control, their optimization objectives are mostly limited to a single indicator, and there is no multi-objective balance of "pose accuracy-energy consumption-joint protection". In practical applications, this may cause the manipulator to excessively rotate the joints to avoid obstacles when grabbing, not only increasing energy consumption, but also possibly causing the joints to be in extreme positions for a long time, shortening the service life.

[0024] In summary, the related visual grabbing technology of manipulators has significant deficiencies in degrees of freedom flexibility, visual perception robustness, control coordination accuracy, and grabbing verification reliability, making it difficult to meet the high-precision and high-stability grabbing requirements in complex environments.

[0025] The embodiment provides a multi-degree-of-freedom mechanical arm vision recognition and grabbing method, first, point cloud data of a target object is acquired by using a depth camera, target feature points are determined by screening the point cloud data, the semantic type of each point is accurately judged by combining a semantic segmentation model, and through preset confidence threshold and mechanical arm workspace depth range constraint, the effective information truly related to the grabbing task is stripped from the mass point cloud of the complex scene. Breakthrough the limitation of traditional visual recognition which only relies on color or simple shape features, even if facing transparent objects, shiny surfaces or irregular shaped target objects, the target can also be locked through the spatial distribution and reflection intensity characteristics of three-dimensional point cloud. Secondly, the target feature points are converted into mechanical arm base coordinate system pose data, realizing accurate mapping from visual perception space to mechanical arm motion space. The conversion matrix obtained based on the calibration board strictly maps the camera coordinate system coordinates output by the depth camera to the mechanical arm base coordinate system, eliminating the spatial deviation between different sensors. Through solving the optimal joint angle based on the coordinate system pose data, the preset multi-target optimization model and the obstacle avoidance constraint condition, the redundancy characteristics of the seven-degree-of-freedom mechanical arm are fully utilized. The multi-target optimization model does not pursue single end pose accuracy, but considers pose accuracy, motion energy consumption and joint limit protection: through the weight coefficient, the priority of different targets is balanced, while ensuring that the end can accurately reach the target position, the intermediate joint attitude is adjusted to avoid obstacles by using the redundant freedom, and the joint motion energy consumption is minimized, and the joint is prevented from being in the limit position for a long time to avoid aggravating wear. Compared with the simple trajectory generation of traditional few-degree-of-freedom mechanical arms, it can adapt to more complex environments and prolong the service life of the mechanical arm, thereby realizing the coordinated optimization of flexibility, safety and economy. Through acquiring the end pose data fed back by the depth camera in real time, the joint angle correction amount is calculated based on a preset formula, and a dynamic closed-loop control mechanism is constructed. In the process of mechanical arm motion, the actual pose of the end is continuously monitored by using the depth camera, the deviation matrix is generated by comparing the theoretical planning pose, and the pose deviation is converted into the joint angle correction amount through the Jacobian matrix. Real-time correction can dynamically compensate the pose drift of the mechanical arm caused by link deformation, load change or environmental interference, so as to ensure that the end effector can accurately track the target position, and the problem of error accumulation in traditional open-loop control is avoided. Then, the PID controller outputs the control signal based on the optimal joint angle and the joint angle correction amount, realizing the smoothness and stability of the mechanical arm motion. Through the synergistic effect of proportional, integral and differential coefficients, the PID control quickly responds to the joint angle correction demand: the proportional term adjusts the deviation immediately, the integral term eliminates the accumulated error, and the differential term suppresses the action overshoot. It ensures that the mechanical arm moves smoothly when adjusting the attitude or approaching the target object, avoids the target object from sliding or the mechanical arm from vibrating caused by motion mutation, and improves the reliability of the grabbing execution.The success of grabbing the target object is verified through point cloud collision detection, objective and quantitative grabbing state judgment criteria are established, the point clouds of the end effector and the target object are segmented, the minimum contact distance and the contact area ratio of the two are calculated, and whether the stable clamping is formed is accurately judged from the three-dimensional space angle. When the contact distance is less than the set threshold and the contact area ratio meets the requirement, it can be determined that the grabbing is successful, otherwise the retry mechanism is triggered. The "virtual covering" and "actual clamping" are effectively distinguished, the misjudgment rate is reduced, and the overall grabbing success rate is further improved through the closed-loop verification and retry mechanism, and the demand for manual intervention is reduced. Through the implementation of the application, the problems that the related mechanical arm visual grabbing technology has significant deficiencies in flexibility, visual perception robustness, control coordination precision and grabbing verification reliability, and is difficult to adapt to the high-precision and high-stability grabbing demand in complex environment are solved.

[0026] According to the embodiment of the application, a multi-degree-of-freedom mechanical arm visual recognition grabbing method embodiment is provided. It should be noted that the steps shown in the flowchart of the drawing can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0027] In this embodiment, a multi-degree-of-freedom mechanical arm visual recognition grabbing method is provided, which is applied to an intelligent low-voltage electrical system, the intelligent low-voltage electrical system includes an image sensor module, a communication module, an intelligent chip module and a low-voltage electrical module, Figure 1 is a flowchart of a multi-degree-of-freedom mechanical arm visual recognition grabbing method according to the embodiment of the application, as Figure 1 shown, the flowchart includes the following steps: Step S101, acquiring point cloud data of the target object by using a depth camera, and screening the point cloud data to determine target feature points.

[0028] Specifically, the above step S101 includes: Step S1011, the depth camera collects point cloud data P of the target object, , is the i-th point, is the three-dimensional coordinate, is the reflection intensity.

[0029] Further, the depth camera emits a detection signal to the space where the target object is located based on active or passive optical principles, and receives the reflection information, and then generates point cloud data P containing spatial position and surface characteristics. The point cloud data P here completely records the scene information in the form of discrete points: represents the i-th spatial sampling point, Precise description of the three-dimensional coordinates of the point in the camera coordinate system is used to locate the spatial position of the target object. As a reflection intensity parameter, it reflects the reflection characteristics of the target object surface to the detection signal. For example, the reflection intensity of a metal surface is usually higher than that of a non-metal, and there is a difference in reflection intensity between smooth and rough surfaces.

[0030] Step S1012, input the point cloud data into the semantic segmentation model to obtain the semantic probability of each point The semantic type corresponding to the semantic probability includes: target object type, background type and obstacle type.

[0031] Further, the semantic segmentation model relies on a deep learning algorithm to classify and infer each point Output semantic probability Map the point cloud data to semantic categories such as "target object type", "background type", and "obstacle type". Break through the dependence on single features such as color and shape in traditional visual recognition. Even if the target object is obscured, the surface is reflective, or the shape is irregular, the semantic segmentation model can capture the global features and local correlations of the point cloud by learning a large amount of labeled data to determine the probability of each point belonging to the target object.

[0032] Step S103, screening the point cloud data based on the screening formula to determine the target feature points, the screening formula is as follows: (1) Wherein, representing the target feature point, representing the confidence threshold, the value range is [0.6, 0.8], representing the depth range of the working space of the seven-degree-of-freedom carbon fiber mechanical arm.

[0033] Further, to accurately extract and grasp the effective information related to the task from the complex scene, the point cloud data after preliminary processing needs to be screened again based on the screening formula. The screening formula A double constraint mechanism is constructed: on the one hand, the semantic probability constraint Ensure that the selected points have high confidence target object attributes, representing the maximum value of the semantic probability in the current point cloud data, multiplied by the confidence threshold , The value is 0.6-0.8, balancing the risk of missed detection and false detection, as the semantic threshold for screening, filtering out background, obstacle and other interference points; on the other hand, the depth range constraint Then combine the physical working space of the seven-degree-of-freedom carbon fiber mechanical arm to limit the depth range of the target feature points, exclude point clouds that the mechanical arm cannot reach or exceed the effective working range, and ensure the feasibility of subsequent motion planning. Through this screening process, the original point cloud data is refined into a point set containing only the key features of the target object .

[0034] Step S102, convert the target feature point into the mechanical arm base coordinate system pose data.

[0035] Specifically, the target feature point is converted into the mechanical arm base coordinate system pose data, and the conversion formula is as follows: (2) Wherein, represents the coordinates of the target feature point in the camera coordinate system, which is directly output by the depth camera, represents the coordinates of the converted target feature point in the mechanical arm base coordinate system, represents the conversion matrix, which is obtained by calibrating the calibration plate and contains the rotation matrix R and the translation vector t.

[0036] Further, the key to establishing the spatial correlation between visual perception and mechanical arm motion control is to realize the coordinate system unity. The target feature point coordinates output by the depth camera are defined based on the camera coordinate system, while the motion planning of the mechanical arm needs to be carried out in its base coordinate system. Due to the differences in installation position and orientation, there is an inherent deviation in spatial pose between the two. To eliminate this deviation, a conversion matrix is introduced to establish the mapping relationship from the camera coordinate system to the mechanical arm base coordinate system through homogeneous coordinate transformation. The construction of the conversion matrix includes two key dimensions: rotation and translation. The rotation matrix describes the attitude difference between the camera coordinate system and the mechanical arm base coordinate system. For example, when the camera is installed at an angle, the rotation matrix needs to compensate for the angular deviation. The translation vector quantifies the spatial position difference between the origins of the two coordinate systems, reflecting the physical distance between the camera and the mechanical arm base. The matrix is obtained by calibrating the calibration plate: place the calibration plate with known geometric dimensions and spatial position in the working scene, and the depth camera collects the camera coordinate system coordinates of the calibration plate feature points. Combined with the theoretical coordinates in the mechanical arm base coordinate system, the optimal parameters of the conversion matrix are solved by least squares method and other algorithms to ensure the accuracy of the coordinate conversion. In the actual conversion process, the three-dimensional coordinates are expanded to four-dimensional homogeneous coordinates by utilizing the mathematical properties of homogeneous coordinates, and the rotation and translation transformation is completed at one time through matrix multiplication. The essence of this operation is to convert the target feature point from the "camera perspective" spatial description to the "mechanical arm perspective" pose data, so that the mechanical arm can understand the spatial position and attitude of the target object based on its kinematic model, and provide a unified spatial reference for subsequent inverse kinematics solving and trajectory planning.

[0037] Step S103, based on the coordinate system pose data, the preset multi-objective optimization model and the obstacle avoidance constraint condition, the optimal joint angle is solved.

[0038] Specifically, in the preset multi-objective optimization model, The optimization objective function is as follows: ​

[0039] wherein the weight coefficient ; The sub-targets are defined as follows: pose accuracy wherein is the desired end-effector pose matrix, is the actual pose matrix, is the F-norm; energy consumption wherein is the joint energy consumption coefficient of the kth joint, which is positively correlated with the joint inertia, is the joint angular velocity of the kth joint; joint limit penalty wherein is the safety threshold value, is the maximum limit angle of the kth joint, is the decay factor, which triggers the penalty when .

[0040] Further, the core of planning a motion from the current state to the target pose for a seven-degree-of-freedom carbon fiber robot arm is to solve the optimal joint angle that satisfies multiple constraints. The seven-degree-of-freedom robot arm has redundant degrees of freedom because the number of joints is more than the six degrees of freedom required to complete the end-effector pose, and the inverse kinematics solution is not unique, providing space for multi-objective optimization. On the basis of meeting the end-effector pose requirements, additional targets such as energy consumption and joint protection can be considered to improve the rationality of the motion and the service life of the robot arm. The preset multi-objective optimization model constructs a comprehensive optimization objective function that includes pose accuracy, motion energy consumption, and joint limit protection. The priority of different targets is balanced by weight coefficients: the pose accuracy sub-target e quantifies the accuracy of the end-effector reaching the target position and attitude by calculating the F-norm of the expected end-effector pose matrix and the actual pose matrix. The F-norm comprehensively considers the overall deviation of the matrix elements, ensuring that the end-effector pose approximates the target in both position and attitude, which is the basic constraint for the robot arm to complete the grasping task. The motion energy consumption sub-target E: the motion energy consumption of the robot arm is positively correlated with the square of the joint angular velocity and the joint inertia, which is modeled by . Wherein is the energy consumption coefficient of the kth joint, the larger the joint inertia, , is the joint angular velocity. This sub-target encourages the robot arm to move in a "low energy consumption" mode, avoiding energy waste and mechanical wear caused by high-speed rotation or frequent start-stop of the joints. The joint limit penalty sub-target is introduced to prevent the joints from being in the limit position for a long time. is the safety threshold value, reserving a 5° buffer space, ​is a decay factor, when the joint angle increases exponentially, forcing the optimization algorithm to adjust the joint angle away from the limit position.

[0041] Specifically, the obstacle avoidance constraint condition is as follows: (4), wherein, is the shortest distance between the link of the seven-degree-of-freedom carbon fiber manipulator and the obstacle, is a safety distance.

[0042] Further, the obstacle avoidance constraint condition further guarantees the motion safety, by calculating the shortest distance between the link of the manipulator and the environmental obstacle, to ensure that it is greater than the safety distance, to avoid collision during the motion.

[0043] Specifically, the above step S103 comprises: The gradient descent method is used to iteratively solve the optimal joint angle , and the formula is as follows: (5) wherein, is a learning rate, is the gradient of the optimization objective function at , and the iteration is converged to .

[0044] Further, to solve the multi-constraint optimization problem, the gradient descent method is used to iteratively search for the optimal joint angle: starting from the initial joint angle , by calculating the gradient of the objective function , the joint angle is gradually updated in the negative direction of the gradient with a learning rate , until the norm of the gradient converges, and finally the optimal joint angle under the multiple constraints of "pose accuracy, reasonable energy consumption, joint safety, and obstacle avoidance" is obtained, which provides direct instructions for the motion control of the manipulator.

[0045] Step S104, acquiring the end pose data fed back by the depth camera in real time, calculating the joint angle correction amount based on a preset formula and the end pose data.

[0046] Specifically, the above step S104 comprises: Step S1041, acquiring the end pose data fed back by the depth camera in real time, assuming that the depth camera detects the actual pose of the end at , is a 4x4 matrix, the theoretical pose of the inverse kinematics planning is , and the deviation matrix is established as follows: ​(6) wherein, is the inverse matrix of the theoretical pose.

[0047] Further, the depth camera continuously monitors the actual pose of the end of the robot arm, at any time , the actual pose matrix of the end is obtained, which contains translation and rotation information. At the same time, according to the initial motion planning, the inverse kinematics algorithm outputs the theoretical pose matrix . In order to quantify the deviation between the actual pose and the theoretical pose, the deviation matrix is constructed by matrix operation: wherein, is the inverse matrix of the theoretical pose matrix, which is used to "reverse convert" the theoretical pose to the reference system of the actual pose, so that the deviation matrix can reflect the translation error and rotation error at the same time. The "difference between the actual pose and the theoretical pose" is converted into a calculable matrix form, which provides a basis for subsequent correction of joint angles.

[0048] Step S1042, the mapping relationship between joint angular velocity and end velocity is described by the robot arm Jacobian matrix , the pose deviation is converted into joint angle correction, and the conversion formula is as follows: (7) wherein, is a 6x7 matrix, is the pseudo-inverse of the Jacobian, is a regularization parameter, which is used to avoid matrix singularity, is the vectorization operation of the deviation matrix, which is used to convert the 4x4 matrix into a 16x1 vector, and the first 6 elements correspond to the translation and rotation deviation.

[0049] Further, the kinematic characteristics of the robot arm are described by the Jacobian matrix , the mapping relationship between joint angular velocity and end linear velocity, angular velocity is established, which reflects the influence of joint micro-motion on the end pose. In order to convert the pose deviation into joint angle correction, the pseudo-inverse of the Jacobian is calculated, wherein, is a regularization parameter, which is used to avoid matrix singularity. When the Jacobian matrix is full rank, direct inversion is easily affected by noise, and regularization can enhance the robustness of the algorithm. Subsequently, the 4x4 deviation matrix is converted into a 16x1 vector by vectorization operation , the first 6 elements are extracted, which correspond to the linearization representation of translation and rotation deviation, and are substituted into the formula , so that the joint angle correction The deviation of the end pose is decomposed into each joint by using the "inverse mapping" of the Jacobian matrix, and the joint angle adjustment amount that minimizes the deviation of the end pose is calculated, so that dynamic closed-loop correction is realized, that is, even if the robot arm deviates from the theoretical trajectory due to external interference in motion, the joint angle can be quickly adjusted through real-time feedback to ensure that the end effector always approaches the target pose.

[0050] In step S105, based on the optimal joint angle and the joint angle correction amount, a control signal is output by a PID controller to control the seven-degree-of-freedom carbon fiber robot arm to grasp the target object.

[0051] Specifically, the above step S105 includes: The corrected joint angle is The control signal output by the PID controller is as follows: (8) Wherein, is a proportional coefficient, is an integral coefficient, is a differential coefficient, , The integral operation is performed on the joint angle deviation.

[0052] Further, the conversion of the joint angle command into the control signal for driving the robot arm to move relies on the PID controller to achieve precise and stable motion control and ensure that the robot arm completes the grasping action according to the expected trajectory. First, the corrected joint angle command is determined. The motion of the robot arm is determined by the initial planned optimal joint angle and the real-time closed-loop correction amount, i.e. This superposition process integrates global optimal planning and local real-time correction: ensures the overall rationality of the robot arm motion, then dynamically compensates for small deviations in the motion process, such as robot arm deformation and environmental interference, to ensure that the end effector always approaches the target pose. Subsequently, the PID controller outputs a control signal for driving the joint motion according to the corrected joint angle command. The core logic of the PID control is to realize precise regulation of joint motion through the synergistic effect of the proportional P, integral I, and differential D terms: the proportional term directly outputs the control amount according to the current joint angle deviation , quickly responds to the deviation, and makes the joint angle approach the target value. The proportional coefficient determines the "sensitivity" of the response, and the larger the value, the faster the correction speed of the deviation, but too large may cause system oscillation. The integral term performs integral operation on the joint angle deviation, accumulates historical deviations, and outputs the control amount, which is used to eliminate "static deviation", such as continuous small deviation caused by friction. The integral coefficient A suitable setting is crucial; too small a setting won't effectively eliminate steady-state error, while too large a setting may lead to integral saturation and overshoot. (Differential term) The control output is based on the rate of change of joint angle deviation, predicting the trend of deviation development and adjusting in advance to suppress system oscillation and improve stability. (Derivative coefficient) This can enhance the damping characteristics of the system and avoid motion overshoot caused by proportional and integral actions. (Through the formula...) The PID controller converts joint angle deviations into continuous control signals, driving the joint motors of the seven-DOF carbon fiber robotic arm. During this process, the lightweight, low inertia, and high rigidity of the carbon fiber robotic arm, combined with the dynamic adjustment capabilities of the PID control, ensures both rapid response of the robotic arm's movements to meet dynamic grasping requirements and stable motion to prevent the target object from slipping due to vibration, ultimately achieving precise and stable grasping of the target object.

[0053] Step S106: Verify whether the seven-DOF carbon fiber robotic arm has successfully grasped the target object through point cloud collision detection.

[0054] Specifically, step S106 includes: Step S1061: After the grasping action is completed, collect the point cloud data of the current scene. , Among them, P grasp This is a collection of point cloud data of the current scene acquired by a depth camera. , is the index of each point in the point cloud data. Let m be the three-dimensional coordinates of the m-th point in the camera coordinate system. Let be the reflection intensity at the m-th point.

[0055] Furthermore, after a brief delay following the completion of the grasping action, to prevent the robotic arm's inertia from affecting the point cloud quality, the depth camera re-acquires point cloud data for the current scene. Unlike the initial point cloud acquisition, this time the scene contains an interactive state of "end-effector + target object": This includes the spatial coordinates and reflection intensity of both. The metal material of the end effector and the surface characteristics of the target object can be determined through... This differentiation provides complete 3D data of the "post-grab state" for subsequent collision detection.

[0056] Step S1062, process the point cloud data After noise reduction and ROI cropping, the end effector point cloud was obtained. Point cloud of target object ; Furthermore, regarding the collected data... Noise reduction and ROI cropping are performed to retain only the spatial range where the end effector and the target object are located, reducing environmental point cloud interference. Through semantic segmentation or geometric feature matching, such as using the CAD model of the end effector and the pre-screened features of the target object, the pre-processed point cloud is segmented into end effector point cloud and target object point cloud . This segmentation process distinguishes the spatial and material characteristics of the two, for example, the regular geometric shape of the end effector, such as the parallel structure of the gripper, and the uniform reflection intensity, which can be distinguished from the irregular shape and diverse reflection characteristics of the target object, providing clear point cloud subsets for subsequent contact analysis.

[0057] Step S1063, the minimum contact distance between the end effector point cloud and the target object point cloud is calculated , as follows: (9) wherein, is any point in , is any point in , and is the Euclidean distance calculation function. Further, the point pairs

[0058] in and are traversed , and the closest distance between the two is calculated by the Euclidean distance formula, reflecting the physical contact tightness between the end effector and the target object. If tends to 0, it indicates direct contact; if the distance is too large, it may not be grasped or the grasp may be loose.

[0059] Step S1064, the contact area ratio of the end effector point cloud and the target object point cloud is calculated , as follows: (10) wherein, is the contact determination threshold.

[0060] Further, the number of points in the end effector point cloud that satisfy the distance ≤ contact determination threshold with the target object point cloud is counted, and the ratio of the total number of points is calculated, i.e. , which quantifies the contact coverage of the end effector and the target object. The higher the ratio, the more complete the contact and the more stable the grasp, such as the larger the contact surface between the gripper and the target object, the more secure the clamping.

[0061] Step S1065, if and , determine that the grasping is successful; otherwise, determine that the grasping is failed.

[0062] Further, in combination with the above two quantitative indicators, a double determination condition is set, and if the grasping is successful, otherwise, the grasping is failed. This avoids virtual contact, such as the end effector only slightly touches the target object, and also eliminates partial contact, such as only the gripper tip contacts, which is easy to cause the target object to slip off, and ensures that the determination result is highly consistent with the actual grasping state.

[0063] In an optional embodiment, the method further comprises: if it is determined that the grasping is failed, triggering a retry mechanism to re-plan the grasping point and the end effector pose.

[0064] Further, if the failure reason is the small contact area, the algorithm will preferentially select the "flat area" or "near-geometric center area" of the target object surface, expand the contact range between the end effector and the target object, and improve ; if the failure reason is the grasping point offset, the algorithm will combine the real-time changes of the point cloud data to recalculate the centroid or feature point coordinates of the target object, adjust the spatial position of the grasping point, and ensure that the end effector can be accurately aligned. If the target object is a planar object, the end effector pose is constrained to be parallel to the normal vector of the target object surface, ensuring that the gripper is in close contact with the target object; if the target object is a cylindrical object, the end effector pose is constrained to be perpendicular to the cylindrical axis, improving the stability of grasping.

[0065] The embodiment provides a multi-degree-of-freedom mechanical arm vision recognition and grabbing method, first, point cloud data of a target object is acquired by using a depth camera, target feature points are determined by screening the point cloud data, the semantic type of each point is accurately judged by combining a semantic segmentation model, and through preset confidence threshold and mechanical arm workspace depth range constraint, the effective information truly related to the grabbing task is stripped from the mass point cloud of the complex scene. Breakthrough the limitation of traditional visual recognition which only relies on color or simple shape features, even if facing transparent objects, shiny surfaces or irregular shaped target objects, the target can also be locked through the spatial distribution and reflection intensity characteristics of three-dimensional point cloud. Secondly, the target feature points are converted into mechanical arm base coordinate system pose data, realizing accurate mapping from visual perception space to mechanical arm motion space. The conversion matrix obtained based on the calibration board strictly maps the camera coordinate system coordinates output by the depth camera to the mechanical arm base coordinate system, eliminating the spatial deviation between different sensors. Through solving the optimal joint angle based on the coordinate system pose data, the preset multi-target optimization model and the obstacle avoidance constraint condition, the redundancy characteristics of the seven-degree-of-freedom mechanical arm are fully utilized. The multi-target optimization model does not pursue single end pose accuracy, but considers pose accuracy, motion energy consumption and joint limit protection: through the weight coefficient, the priority of different targets is balanced, while ensuring that the end can accurately reach the target position, the intermediate joint attitude is adjusted to avoid obstacles by using the redundant freedom, and the joint motion energy consumption is minimized, and the joint is prevented from being in the limit position for a long time to avoid aggravating wear. Compared with the simple trajectory generation of traditional few-degree-of-freedom mechanical arms, it can adapt to more complex environments and prolong the service life of the mechanical arm, thereby realizing the coordinated optimization of flexibility, safety and economy. Through acquiring the end pose data fed back by the depth camera in real time, the joint angle correction amount is calculated based on a preset formula, and a dynamic closed-loop control mechanism is constructed. In the process of mechanical arm movement, the actual pose of the end is continuously monitored by using the depth camera, the deviation matrix is generated by comparing the theoretical planning pose, and the pose deviation is converted into the joint angle correction amount through the Jacobian matrix. Real-time correction can dynamically compensate the pose drift of the mechanical arm caused by link deformation, load change or environmental interference, so as to ensure that the end effector can accurately track the target position, and the problem of error accumulation in traditional open-loop control is avoided. Then, the PID controller outputs the control signal based on the optimal joint angle and the joint angle correction amount, realizing the smoothness and stability of the mechanical arm movement. Through the synergistic effect of proportional, integral and differential coefficients, the PID control quickly responds to the joint angle correction demand: the proportional term adjusts the deviation immediately, the integral term eliminates the accumulated error, and the differential term suppresses the action overshoot. It ensures that the mechanical arm moves smoothly when adjusting the attitude or approaching the target object, avoids the target object from sliding or the mechanical arm from vibrating caused by sudden movement, and improves the reliability of the grabbing execution.Whether the target object is successfully grabbed is verified through point cloud collision detection, an objective and quantitative judgment standard of the grabbing state is established, the point clouds of the end effector and the target object are segmented, the minimum contact distance and the contact area ratio of the two are calculated, and whether a stable clamping is formed is accurately judged from the perspective of three-dimensional space. When the contact distance is less than the set threshold and the contact area ratio meets the requirement, it can be determined that the grabbing is successful, otherwise the retry mechanism is triggered. The "virtual covering" and "actual clamping" are effectively distinguished, the misjudgment rate is reduced, and the overall success rate of grabbing is further improved through the closed-loop verification and retry mechanism, thereby reducing the need for manual intervention. Through the implementation of the application, the problems that the related mechanical arm visual grabbing technology has significant deficiencies in flexibility, visual perception robustness, control coordination accuracy and grabbing verification reliability, and is difficult to adapt to the high-precision and high-stability grabbing requirements in complex environments are solved.

[0066] Although the embodiments of the present application are described in conjunction with the drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes fall within the scope defined by the appended claims.

Claims

1. A method for visual recognition and grasping based on a multi-degree-of-freedom robot arm, applied to a multi-degree-of-freedom robot arm, the multi-degree-of-freedom robot arm comprising a seven-degree-of-freedom carbon fiber robot arm, a depth camera and a controller, the depth camera being installed on a base of the seven-degree-of-freedom carbon fiber robot arm, characterized in that, The method comprises: acquiring point cloud data of a target object by using a depth camera, screening the point cloud data to determine target feature points; converting the target feature points into mechanical arm base coordinate system pose data; solving optimal joint angles based on the coordinate system pose data, a preset multi-target optimization model, and an obstacle avoidance constraint condition; acquiring end pose data fed back by the depth camera in real time, calculating a joint angle correction amount based on a preset formula and the end pose data; outputting a control signal by a PID controller based on the optimal joint angles and the joint angle correction amount to control the seven-degree-of-freedom carbon fiber mechanical arm to grasp the target object; verifying whether the seven-degree-of-freedom carbon fiber mechanical arm successfully grasps the target object by point cloud collision detection.

2. The method of claim 1, wherein, The method of acquiring point cloud data of a target object by using a depth camera, screening the point cloud data to determine target feature points comprises: The depth camera collects point cloud data P of the target object, , is the i-th point, is a three-dimensional coordinate, is a reflection intensity, Input the point cloud data into a semantic segmentation model to obtain a semantic probability of each point The semantic type corresponding to the semantic probability comprises a target object type, a background type and an obstacle type. screening the point cloud data based on a screening formula to determine target feature points, the screening formula being as follows: （1） Wherein, Characterize the target feature points, Characterize the confidence threshold, the value range is 【0.6, 0.8】, Characterize the workspace depth range of the seven-degree-of-freedom carbon fiber mechanical arm.

3. The method of claim 2, wherein, The conversion formula for converting the target feature points into mechanical arm base coordinate system pose data is as follows: （2） wherein, characterizing the coordinates of the target feature points in the camera coordinate system, directly output by the depth camera, characterizing the coordinates of the target feature points in the robot base coordinate system after the transformation, characterizing the transformation matrix, obtained by calibration with the calibration plate, comprising a rotation matrix R and a translation vector t.

4. The method of claim 3, wherein, In the preset multi-target optimization model, the optimization objective function is as follows: ； wherein the weight coefficient ; the sub-object is defined as follows: pose accuracy wherein, is the desired end pose matrix, is the actual pose matrix, is the F-norm; Energy consumption wherein, is the joint energy consumption coefficient of the kth joint, which is positively related to the joint inertia, is the joint angular velocity of the kth joint. joint limit penalty wherein, is a safety threshold, is the maximum limit angle of the k-th joint, is a decay factor, when the penalty is triggered.

5. The method of claim 4, wherein, the obstacle avoidance constraint condition is as follows: （4）, wherein, is the shortest distance of the link of the seven-degree-of-freedom carbon fiber robot arm to the obstacle, is the safety distance.

6. The method of claim 5, wherein, The method of solving optimal joint angles based on the coordinate system pose data, a preset multi-target optimization model, and an obstacle avoidance constraint condition comprises: The optimal joint angles are solved iteratively using a gradient descent method The formula is as follows: （5） wherein, is a learning rate, is the gradient of the optimization objective function at is iterated until convergence.

7. The method of claim 6, wherein, The method of acquiring end pose data fed back by the depth camera in real time, calculating a joint angle correction amount based on a preset formula and the end pose data comprises: The end position data of real-time feedback of the depth camera is acquired, and the depth camera detects the actual end position as at the moment , is a 4x4 matrix, and the theoretical position of inverse kinematics planning is . The deviation matrix is established as follows: （6） wherein, is the inverse matrix of the theoretical pose; Through the Jacobian matrix of the robot arm The mapping relationship between joint angular velocity and end velocity is described, and the pose deviation is converted into joint angle correction amount. The conversion formula is as follows: （7） where, is a 6x7 matrix, is the Jacobian pseudo-inverse, is a regularization parameter to avoid matrix singularity, is a vectorization operation of the bias matrix to convert a 4x4 matrix into a 16x1 vector, taking the first 6 elements corresponding to the translation and rotation bias.

8. The method of claim 7, wherein, The method of outputting a control signal by a PID controller based on the optimal joint angles and the joint angle correction amount to control the seven-degree-of-freedom carbon fiber mechanical arm to grasp the target object comprises: The corrected joint angle is The control signal output by the PID controller is as follows: （8） wherein is a proportional coefficient, is an integral coefficient, is a derivative coefficient, , The joint angle deviation is integrated.

9. The method of claim 8, wherein, The method of verifying whether the seven-degree-of-freedom carbon fiber mechanical arm successfully grasps the target object by point cloud collision detection comprises: After the grabbing action is completed, point cloud data of the current scene is collected ; wherein P grasp is a current scene point cloud data set collected by a depth camera, is an index of each point in the point cloud data, is a three-dimensional coordinate of the mth point in the camera coordinate system is the reflection intensity of the mth point; To the point cloud data After denoising and ROI cropping, the end effector point cloud is segmented With the target point cloud ; Computing end effector point cloud Minimum contact distance with target object point cloud Minimum contact distance with target object point cloud , as follows: （9） wherein is any point in is any point in is the Euclidean distance computation function; Computing end effector point cloud Contact area proportion with target object point cloud Contact area proportion with target object point cloud , as follows: （10） wherein is a contact determination threshold value; If and , determine that the grabbing is successful; otherwise, determine that the grabbing is unsuccessful.

10. The method of claim 9, wherein, The method further comprises: if it is determined that the grasping fails, triggering a retry mechanism to re-plan a grasping point and an end effector pose.

Images