Bionic actuator microspur pickup regulation and control strategy based on vision field feedback

Abstract

The invention provides a bionic actuator microspur pickup regulation and control strategy based on vision field feedback, and belongs to the field of robot control. The problem of low grabbing precision of the robot is solved; the method specifically comprises the following steps that S1, robot parameter information and visual image information are obtained; s2, constructing a space model; searching the grabbed object according to the visual image information, and acquiring the position of the grabbed object in the spatial model to obtain a grabbed position; s3, the robot is moved, the initial position is obtained, and a robot grabbing path is obtained according to the initial position and the grabbing position; according to the grabbing path of the robot and the parameter information of the robot, the grabbing behavior of the robot is controlled; s4, checking and judging the object grabbed by the robot according to the information of the grabbed object; the robot is controlled, and an object is accurately grabbed in an image processing mode; and the grabbed object is inspected, and the grabbing precision of the robot is improved.

Description

Technical Field

[0001] A micro-pickup regulation strategy for a bionic actuator based on visual field feedback according to the present invention relates to the field of robot control. Background Art

[0002] Existing micro-pickup regulation strategies for bionic actuators based on visual field feedback have the following deficiencies:

[0003] Limited application range: Existing precise grasping control methods for robots mainly involve grasping control of specific objects within a fixed range by a robotic arm. The grasping range is small, and the grasping objects are limited; the generalization ability for complex scenarios is insufficient;

[0004] Limitations in grasping strategies: Traditional methods rely on predefined grasping templates (such as fixed angles of parallel grippers, fixed adsorption surfaces of suction cups), and it is difficult to adapt to changes in the shape, material, or posture of the target object. For example, when grasping an object placed obliquely, the fixed grasping angle cannot fit the target surface;

[0005] Insufficient coordination of robotic arm segments: During the movement of a multi-joint robotic arm, dynamic synchronization cannot be achieved between the segments (joints), resulting in a decrease in the movement accuracy of the end effector (such as a gripper, tool), reducing the grasping accuracy;

[0006] Inadequate consideration of special situations: A three-dimensional coordinate acquisition method is disclosed in the patent application document with the publication number CN112372641A. The coordinates of an object are obtained through internal parameter calibration and hand-eye calibration of a global camera and a camera at the end of the robotic arm. Obtaining coordinates only through the camera is prone to parameter loss in some special situations (such as too low or too high brightness), resulting in errors and low accuracy of the three-dimensional model. Summary of the Invention

[0007] Aiming at the deficiencies of the existing technology, the purpose of the present invention is to provide a micro-pickup regulation strategy for a bionic actuator based on visual field feedback, aiming to solve the problem of complex robot control.

[0008] To achieve the above purpose, the present invention is realized through the following technical solutions:

[0009] A micro-pickup regulation strategy for a bionic actuator based on visual field feedback includes:

[0010] Step S1: Obtain robot parameter information and visual image information;

[0011] Step S2: According to the visual image information, construct a spatial model; obtain grasping object information, search for the grasping object in combination with the visual image information, and obtain the position of the grasping object in the spatial model to obtain the grasping position;

[0012] Step S3: Move the robot based on the spatial model and the grasping position, obtain the current position of the robot to get the initial position, and obtain the robot's grasping path according to the initial position and the grasping position; Detect the adjacent objects of the grasping object according to the spatial model, obtain the gap between the grasping object and the adjacent objects, calculate the grasping angle during grasping in combination with the robot parameter information and the grasping path, detect the robot's grasping path according to the grasping angle and the gap between the grasping object and the adjacent objects, judge the grasping behavior based on the detection result, and control the grasping behavior of the robot;

[0013] Step S4: Obtain the features of the actual grasping object of the robot; Combine the grasping object information to check and judge the actual grasping object of the robot.

[0014] Further, the specific steps of the said Step S1 are as follows:

[0015] Step S11: Obtain the number of joints n of the robot's manipulator; Measure the length cd of the manipulator segments according to the number of joints of the manipulator, count the lengths of the manipulator segments to obtain the manipulator segment list lbc, lbc = [cd(1), cd(2),..., cd(n - 1)];

[0016] Step S12: Obtain the pixel information of the visual image of the space where the robot is located, set the upper left corner of the visual image as the image origin, set the horizontal right direction as the horizontal axis, and set the vertical downward direction as the vertical axis to construct a pixel coordinate system; Represent the pixel position in the visual image with (u, v); where (u, v) represents the pixel point at the u-th row and v-th column from the image origin; Obtain the depth information of the visual image; The pixel information of the visual image and the depth information of the visual image constitute the visual image information.

[0017] Further, the specific steps of the said Step S2 are as follows:

[0018] Step S21: Represent the visual image in three dimensions from the visual image information to obtain the three-dimensional data of the visual image, and construct a spatial model according to the three-dimensional data of the visual image;

[0019] Step S22: Obtain the grasping object information; Compare the visual image according to the grasping object information to obtain the image position of the grasping object, and obtain the spatial position of the grasping object in the spatial model according to the image position of the grasping object to get the grasping position (Zx, Zy, Zz).

[0020] Further, the specific steps of the said Step S21 are as follows:

[0021] Step S211: Obtain the current position as the origin position (0, 0, 0), define the horizontal right direction of the current view as the x-axis, and define the horizontal downward direction of the current view as the y-axis; define the direction outward along the optical axis as the z-axis; construct a three-dimensional coordinate system according to the origin position, x-axis, y-axis, and z-axis;

[0022] Step S212: According to the visual image information and the three-dimensional coordinate system, calculate the three-dimensional coordinates of the image pixels, and construct a space model from the three-dimensional coordinates of the image pixels.

[0023] Further, the specific steps of step S212 are as follows:

[0024] From the visual image information, obtain the depth information of the visual image. According to the depth information of the visual image, obtain the depth corresponding to the image pixels, and get the coordinate Sz corresponding to the image pixels in the z-axis direction;

[0025] According to the intersection position of the pixel coordinate system and the optical axis, obtain the optical center coordinates (Cx, Cy); obtain the focal lengths (Fx, Fy);

[0026] According to the pixel information of the visual image, obtain the position u corresponding to the image pixels in the horizontal direction of the pixel coordinate system. Combine the horizontal coordinate Cx of the optical center coordinates, the horizontal coordinate Fx of the focal length, and the coordinate Sz corresponding to the image pixels in the z-axis to calculate the coordinate Sx corresponding to the image pixels in the x-axis direction;

[0027] According to the pixel information of the visual image, obtain the position v corresponding to the image pixels in the vertical direction of the pixel coordinate system. Combine the vertical coordinate Cy of the optical center coordinates, the vertical coordinate Fy of the focal length, and the coordinate Sz corresponding to the image pixels in the z-axis to calculate the coordinate Sy corresponding to the image pixels in the y-axis direction;

[0028] According to the coordinate Sz corresponding to the image pixels in the z-axis direction, the coordinate Sx corresponding to the image pixels in the x-axis direction, and the coordinate Sy corresponding to the image pixels in the y-axis direction, obtain the three-dimensional coordinates (Sx, Sy, Sz) of the image pixels; according to the three-dimensional coordinates of the image pixels, fill the space to obtain a space model;

[0029] Obtain the point cloud data of the object in the space. According to the point cloud data, obtain the object point coordinates (dbx, dby, dbz) in the point cloud data and the reflected light intensity corresponding to the point coordinates. According to the object point coordinates (dbx, dby, dbz) in the point cloud data, obtain the reflected light intensity of the corresponding point coordinates in the space model. Compare the reflected light intensity corresponding to the point coordinates in the point cloud data with the reflected light intensity of the corresponding point coordinates in the space model. If the two reflected light intensities are different, replace the object in the space model according to the object point coordinates of the point cloud data, and optimize the space model.

[0030] Further, the specific steps of step S3 are as follows:

[0031] Step S31: Move the robot according to the spatial model and the grasping position, move the robot towards the grasping position. When the robot cannot move, obtain the three-dimensional coordinates of the current position of the robot to get the initial position (Jx, Jy, Jz); according to the initial position and the grasping position, take the initial position as the starting position and the grasping position as the target position, connect the starting position and the target position to obtain the grasping vector (Lx, Ly, Lz), and the grasping vector (Lx, Ly, Lz) = (Zx, Zy, Zz) - (Jx, Jy, Jz), and take the grasping vector as the grasping path of the robot;

[0032] Step S32: According to the robot parameter information, obtain the list of manipulator arm segments lbc, lbc = [cd(1), cd(2),..., cd(n - 1)]; combine the robot grasping path to judge the grasping behavior;

[0033] Step S33: Fit the manipulator arm segments to the robot grasping path to obtain the grasping posture of the manipulator arm segments, and grasp the grasping object according to the grasping posture.

[0034] Further, the specific steps of step S32 are as follows:

[0035] Step S321: Calculate the maximum reachable distance of the manipulator arm according to the list of manipulator arm segments lbc = [cd(1), cd(2),..., cd(n - 1)] to obtain the maximum distance ZCJ:

[0036] ZCJ = cd(1) + cd(2) +... + cd(n - 1);

[0037] Step S322: According to the robot grasping path, obtain the grasping vector (Lx, Ly, Lz), and calculate the distance between the robot and the grasping object according to the grasping vector to obtain the grasping distance ZQJ:

[0038]

[0039] Step S323: Judge whether the robot can grasp according to the maximum distance ZCJ and the grasping distance ZQJ:

[0040] If ZCJ ≥ ZQJ, it indicates that the robot can grasp the grasping object; enter step S33;

[0041] If ZCJ < ZQJ, it indicates that the robot cannot grasp the grasping object, and an alarm error is given.

[0042] Further, the specific steps of step S33 are as follows:

[0043] Step S331: Obtain the real-time position of the robotic arm segment, and calculate the projections of the robotic arm segment on the three-dimensional coordinate system according to the real-time position. The specific calculation is as follows:

[0044] Obtain the joint coordinates ZB1(Z1x, Z1y, Z1z) and ZB2(Z2x, Z2y, Z2z) adjacent to the robotic arm segment, and calculate the projection vector TY(Z1x - Z2x, Z1y - Z2y, Z1z - Z2z) according to the joint coordinates ZB1(Z1x, Z1y, Z1z) and ZB2(Z2x, Z2y, Z2z);

[0045] Obtain the normal vector FXL(XAx, XAy, XAz) in the x-axis direction, and calculate the projection of the projection vector TY in the x-axis direction according to the projection vector TY and the normal vector FXL of the x-axis to obtain the projection distance xty on the x-axis;

[0046] Obtain the normal vector FYL(YAx, YAy, YAz) in the y-axis direction, and calculate the projection of the projection vector TY in the y-axis direction according to the projection vector TY and the normal vector FYL of the y-axis to obtain the projection distance yty on the y-axis;

[0047] Obtain the normal vector FZL(ZAx, ZAy, ZAz) in the z-axis direction, and calculate the projection of the projection vector TY in the z-axis direction according to the projection vector TY and the normal vector FZL of the z-axis to obtain the projection distance zty on the z-axis;

[0048] Step S332: Accumulate the projection distances of the robotic arm segment on the x-axis to obtain the total projection distance zxt on the x-axis; accumulate the projection distances of the robotic arm segment on the y-axis to obtain the total projection distance zyt on the y-axis; accumulate the projection distances of the robotic arm segment on the z-axis to obtain the total projection distance zzt on the z-axis, and combine the grasping vector (Lx, Ly, Lz) to move the robotic arm segment;

[0049] When the projection distance is less than the distance in the corresponding direction of the grasping vector, move in that direction;

[0050] When the projection distance is greater than the distance in the corresponding direction of the grasping vector, move in the opposite direction;

[0051] When the total projection distances zxt, zyt, zzt of the robotic arm segment on the x-axis, y-axis, and z-axis are equal to the values of the grasping vector (Lx, Ly, Lz), obtain the grasping posture of the robotic arm, and grasp the grasping object according to the grasping posture.

[0052] Furthermore, the specific steps of step S332 are as follows:

[0053] Obtain the grasping surface of the object to be grasped, extract the vertices of the grasping surface, and record the number of vertices as g; obtain the coordinates dz (dzx, dzy, dzz) of the vertices to get dz1, dz2, …, dzg; calculate the central coordinate zxb (ZXx, ZXy, ZXz) of the grasping surface according to the coordinates dz of the vertices:

[0054]

[0055] Connect the central coordinate with any two vertices to obtain vectors xla and xlb; perform a cross product calculation on vectors xla and xlb to obtain the vertical vector zcz (CZx, CZy, CZz) of the grasping surface;

[0056] According to the vertical vector zcz (CZx, CZy, CZz) of the grasping surface, calculate its angles θx, θy, θz with the x-axis, y-axis, and z-axis;

[0057] Obtain the position of the end effector of the robot, move the position of the end effector of the robot to the central coordinate zxb (ZXx, ZXy, ZXz) of the grasping surface, adjust the angle of the end effector of the robot so that its angle with the x-axis is θx, its angle with the y-axis is θy, and its angle with the z-axis is θz, and perform grasping;

[0058] Obtain the adjacent objects of the object to be grasped, and obtain the coordinates Lzb (LZx, LZy, LZz) of the adjacent objects; according to the coordinates Lzb (LZx, LZy, LZz) of the adjacent objects, obtain the coordinates of two vertices of the object to be grasped that are close to it, connect the vertex coordinates to obtain an adjacent side; map the coordinates Lzb (LZx, LZy, LZz) of the adjacent objects to the adjacent side to obtain a mapping point YSD (YSx, YSy, YSz), and calculate the distance between the adjacent object and the adjacent side of the object to be grasped according to the mapping point YSD (YSx, YSy, YSz) and the coordinates Lzb (LZx, LZy, LZz) of the adjacent objects to obtain a clearance value JXZ;

[0059]

[0060] According to the coordinates Lzb (LZx, LZy, LZz) of the adjacent objects and the central coordinate zxb (ZXx, ZXy, ZXz) of the grasping surface, obtain a collision vector, and calculate the angles of the collision vector to obtain αx, αy, αz;

[0061] Calculate the distance between the coordinates Lzb (LZx, LZy, LZz) of the adjacent object and the central coordinates zxb (ZXx, ZXy, ZXz) of the grasping surface to obtain the grasping depth zsd;

[0062]

[0063] Obtain the real-time position of the robot end effector. According to the real-time position coordinates of the robot end effector, obtain the distance vector Mjl (MLx, Mly, MLz) between the robot end effector and the coordinates Lzb (LZx, LZy, LZz) of the adjacent object; calculate according to the distance vector Mjl to obtain the collision distance pzl;

[0064]

[0065] Obtain the position of the current robot end effector, and move it towards the adjacent object according to the collision distance pzl. When the collision distance pzl = 0, if the distance between the position of the robot end effector and the adjacent object is greater than the grasping depth zsd, it is determined that grasping can be performed.

[0066] Further, the step S332 further includes:

[0067] Perform vector summation on the collision vector and the distance vector to obtain the movement vector ydx (YDx, YDy, YDz) of the robot end effector. Using the movement vector as the center line, detect the obstacle to obtain the obstacle coordinates; according to the obstacle coordinates and the central coordinates zxb (ZXx, ZXy, ZXz) of the grasping surface, construct the obstacle vectors zaw1, zaw2,..., zaws; calculate according to the movement vector ydx (YDx, YDy, YDz) and the obstacle vectors zaw1, zaw2,..., zaws to obtain the change angle of the robot end effector;

[0068] Calculate the included angles of the movement vectors of the robot end effector to obtain the real-time angles βx, βy, and βz; obtain the grasping angles θx, θy, and θz of the robot end effector; based on the real-time angles βx, βy, and βz and the grasping angles θx, θy, and θz, obtain the movement angles of the robot end effector, and compare the movement angles of the robot end effector with the change angles of the robot end effector. If the movement angle of the robot end effector is less than the change angle of the robot end effector, it is determined that movement can be performed; obtain the real-time angles of the robot end effector, the width mkd and the length mcd of the robot end effector; make a judgment in combination with the grasping angles θx, θy, and θz of the robot end effector; when within the change angles bhx, bhy, and bhz of the robot end effector, the angles of the robot end effector conform to the grasping angles θx, θy, and θz, and at the same time the width mkd of the robot end effector is less than the clearance value JXZ and the length mcd of the robot end effector is greater than the grasping depth zsd; it is determined that grasping can be performed.

[0069] Compared with the prior art, the beneficial effects of the present invention are:

[0070] Expand the application scope: For the three-dimensional space model of the grasping range space framework, grasp the items within the space according to the grasping object, realize dynamic large-range grasping, and realize the generalization of robot grasping in multiple fields;

[0071] Optimize the grasping strategy: Conduct a specific analysis of the grasping object, and adjust the robotic arm according to the central position and offset angle of the grasping object, so that the robotic arm grasping can adapt to different shape or posture changes, and enhance the grasping stability and applicability;

[0072] Synchronously adjust the robotic arm segments: Divide the robotic arm segments into separate individuals for motion control, refine the original motion trajectory of the robotic arm, optimize the grasping speed, decompose the movement trajectory of the robotic arm, determine the movement trajectory of each robotic arm segment, and determine the specific grasping path;

[0073] Optimize model construction: A three-dimensional coordinate acquisition method is disclosed in the patent application document with the publication number CN112372641A. On this basis, the present invention introduces a lidar to scan the objects in the space to obtain the point cloud data of the objects in the space, combines the point cloud data with the original model, and guards against special situations with too low or too high brightness; optimize model construction to make the model more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] By reading the following detailed description of the non-restrictive embodiments with reference to the accompanying drawings, other features, objects, and advantages of the present invention will become more apparent:

[0075] Figure 1Schematic diagram of the method of the present invention;

[0076] Figure 2 Schematic diagram of the main data processing of the present invention;

[0077] Figure 3 Schematic diagram of the grasping of the present invention; Detailed implementation manners

[0078] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific implementation manners.

[0079] Example 1

[0080] Please refer to Figure 1 , a bionic actuator micro-pickup control strategy based on visual field feedback includes:

[0081] It should be noted that: a bionic actuator refers to an intelligent device that imitates the motion mechanism or sensing function of a living being, and in the present invention, it is a robot.

[0082] Step S1: Obtain robot parameter information and visual image information;

[0083] Step S11: Obtain the number of joints n of the robot manipulator; measure the length cd of the manipulator arm segments according to the number of joints of the manipulator arm, and count the lengths of the manipulator arm segments to obtain a manipulator arm segment list lbc, lbc = [cd(1), cd(2),..., cd(n - 1)];

[0084] It should be noted that: a manipulator arm segment refers to a rigid structure part in the manipulator arm that connects two adjacent joints and is responsible for transmitting motion and force; its number is the number of joints minus 1; cd(1) represents the length of the first manipulator arm segment;

[0085] Step S12: Use a depth camera to obtain the pixel information of the visual image in the space where the robot is located. Set the upper left corner of the visual image as the image origin, set the horizontal right direction as the horizontal axis, and the vertical downward direction as the vertical axis to construct a pixel coordinate system; represent the pixel position in the visual image by (u, v); where (u, v) represents the pixel point at the u-th row and v-th column from the image origin; the depth camera obtains the depth information of the visual image; the pixel information of the visual image and the depth information of the visual image constitute the visual image information.

[0086] It should be noted that: A depth camera is a device that can obtain the depth information of objects in a scene. It measures the distance between the object and the camera and constructs a three-dimensional space model. Compared with traditional two-dimensional cameras, depth cameras can provide richer spatial information and are widely used in fields such as three-dimensional reconstruction, human-computer interaction, robot navigation, augmented reality (AR), virtual reality (VR), and autonomous driving.

[0087] Step S2: Construct a spatial model based on the visual image information; obtain the information of the grasping object, search for the grasping object in combination with the visual image information, and obtain the grasping position by obtaining the position of the grasping object in the spatial model.

[0088] Step S21: Represent the visual image in three dimensions based on the visual image information to obtain the three-dimensional data of the visual image, and construct a spatial model according to the three-dimensional data of the visual image. The specific construction process is as follows:

[0089] Step S211: Obtain the position where the depth camera is located as the origin position (0, 0, 0), define the horizontal right direction of the camera view as the x-axis, and define the horizontal downward direction of the camera view as the y-axis; define the direction outward along the camera optical axis as the z-axis; construct a three-dimensional coordinate system according to the origin position, x-axis, y-axis, and z-axis.

[0090] Step S212: Calculate the three-dimensional coordinates of the image pixels according to the visual image information and the three-dimensional coordinate system, and construct a spatial model from the three-dimensional coordinates of the image pixels.

[0091] Step S2121: Obtain the depth information of the visual image from the visual image information, and obtain the depth corresponding to the image pixel according to the depth information of the visual image to obtain the coordinate Sz corresponding to the image pixel in the z-axis direction.

[0092] Step S2122: Obtain the optical center coordinates (Cx, Cy) according to the intersection position of the pixel coordinate system and the camera optical axis; obtain the focal lengths (Fx, Fy) according to the depth camera.

[0093] It should be noted that: The focal length represents the ability of the camera lens to focus the light in the three-dimensional space onto the image plane, and is in pixels.

[0094] According to the pixel information of the visual image, obtain the position u corresponding to the image pixel in the horizontal direction of the pixel coordinate system. Combine the horizontal coordinate Cx of the optical center coordinates, the horizontal coordinate Fx of the focal length, and the coordinate Sz corresponding to the image pixel in the z-axis to calculate the coordinate Sx corresponding to the image pixel in the x-axis direction.

[0095]

[0096] According to the pixel information of the visual image, obtain the position v corresponding to the image pixel in the vertical direction of the pixel coordinate system. Combine the vertical coordinate Cy of the optical center, the vertical coordinate Fy of the focal length, and the coordinate Sz corresponding to the image pixel on the z-axis to calculate the coordinate Sy corresponding to the image pixel in the y-axis direction.

[0097]

[0098] Step S2123: According to the coordinate Sz corresponding to the image pixel in the z-axis direction, the coordinate Sx corresponding to the image pixel in the x-axis direction, and the coordinate Sy corresponding to the image pixel in the y-axis direction, obtain the three-dimensional coordinates (Sx, Sy, Sz) of the image pixel. According to the three-dimensional coordinates of the image pixel, fill the space to obtain a space model.

[0099] Step S2124: Use a lidar to scan the objects in the space to obtain the point cloud data of the objects in the space. Check the space model according to the point cloud data as follows:

[0100] From the point cloud data, obtain the object point coordinates (dbx, dby, dbz) in the point cloud data and the reflected light intensity corresponding to the point coordinates. According to the object point coordinates (dbx, dby, dbz) in the point cloud data, obtain the reflected light intensity of the corresponding point coordinates in the space model. Compare the reflected light intensity corresponding to the point coordinates in the point cloud data with the reflected light intensity of the corresponding point coordinates in the space model. If the reflected light intensities of the two are different, replace the object in the space model according to the object point coordinates of the point cloud data and optimize the space model.

[0101] It should be noted that: The reflected light intensity of the lidar is more accurate and less affected by ambient light interference, making it suitable for long-distance detection (up to hundreds of meters). Its high resolution and anti-interference ability enable it to perform excellently in complex environments.

[0102] Optimize the point cloud registration of the optimized space model and the point cloud data through a feature matching algorithm to eliminate the spatial offset between devices. Fill the hole areas caused by occlusion or reflection in the depth camera according to the lidar point cloud, and constrain the noise points of the lidar point cloud according to the dense data of the depth camera to improve the model accuracy.

[0103] Step S22: Obtain the information of the grasping object; according to the information of the grasping object, compare the visual image to obtain the image position of the grasping object, and obtain the spatial position of the grasping object in the space model according to the image position of the grasping object to obtain the grasping position (Zx, Zy, Zz).

[0104] It should be noted that: The information of the grasping object refers to the color and shape of the grasping object.

[0105] Step S3: Move the robot based on the spatial model and the grasping position, obtain the current position of the robot to get the initial position, and obtain the robot grasping path based on the initial position and the grasping position; Detect the adjacent objects of the grasping object according to the spatial model, obtain the gap between the grasping object and the adjacent objects, calculate the grasping angle during grasping in combination with the robot parameter information and the grasping path, detect the robot grasping path according to the grasping angle and the gap between the grasping object and the adjacent objects, judge the grasping behavior based on the detection result, and control the grasping behavior of the robot;

[0106] Step S31: Move the robot based on the spatial model and the grasping position, move the robot towards the grasping position. When the robot cannot move, obtain the three-dimensional coordinates of the current position of the robot to get the initial position (Jx, Jy, Jz); Based on the initial position and the grasping position, use the initial position as the starting position and the grasping position as the target position, connect the starting position and the target position to obtain the grasping vector (Lx, Ly, Lz), and the grasping vector (Lx, Ly, Lz) = (Zx, Zy, Zz) - (Jx, Jy, Jz). Use the grasping vector as the robot grasping path;

[0107] It should be noted that: The grasping vector is the connection between the initial position and the grasping position, and the grasping vector represents the shortest distance between the initial position and the grasping position.

[0108] Step S32: Obtain the list of robotic arm segments lbc according to the robot parameter information, lbc = [cd(1), cd(2),..., cd(n - 1)]; Combine the robot grasping path to judge the grasping behavior;

[0109] Step S321: Calculate the maximum reachable distance of the robotic arm according to the list of robotic arm segments lbc = [cd(1), cd(2),..., cd(n - 1)] to obtain the maximum distance ZCJ:

[0110] ZCJ = cd(1) + cd(2) +... + cd(n - 1);

[0111] It should be noted that: When all the robotic arm segments are in the same straight line direction, the reachable distance of the robotic arm is the longest.

[0112] Step S322: Obtain the grasping vector (Lx, Ly, Lz) according to the robot grasping path, calculate the distance between the robot and the grasping object according to the grasping vector to obtain the grasping distance ZQJ:

[0113]

[0114] Step S323: Determine whether the robot can grasp based on the longest distance ZCJ and the grasping distance ZQJ:

[0115] If ZCJ ≥ ZQJ, it indicates that the robot can grasp the grasping object; proceed to step S33;

[0116] If ZCJ < ZQJ, it indicates that the robot cannot grasp the grasping object, and an alarm for error is given.

[0117] Step S33: Fit the robotic arm segments to the robot's grasping path to obtain the grasping posture of the robotic arm segments, and grasp the grasping object according to the grasping posture;

[0118] Step S331: Obtain the real-time position of the robotic arm segments, and calculate the projections of the robotic arm segments on the three-dimensional coordinate system respectively according to the real-time position of the robotic arm segments. The specific calculation is as follows:

[0119] Obtain the joint coordinates ZB1(Z1x, Z1y, Z1z) and ZB2(Z2x, Z2y, Z2z) adjacent to the robotic arm segments, and calculate the projection vector TY(Z1x - Z2x, Z1y - Z2y, Z1z - Z2z) according to the joint coordinates ZB1(Z1x, Z1y, Z1z) and ZB2(Z2x, Z2y, Z2z);

[0120] Obtain the normal vector FXL(XAx, XAy, XAz) in the x-axis direction, and calculate the projection of the projection vector TY in the x-axis direction according to the projection vector TY and the normal vector FXL of the x-axis to obtain the projection distance xty on the x-axis;

[0121] xty = TY · FXL;

[0122] Obtain the normal vector FYL(YAx, YAy, YAz) in the y-axis direction, and calculate the projection of the projection vector TY in the y-axis direction according to the projection vector TY and the normal vector FYL of the y-axis to obtain the projection distance yty on the y-axis;

[0123] yty = TY · FYL;

[0124] Obtain the normal vector FZL(ZAx, ZAy, ZAz) in the z-axis direction, and calculate the projection of the projection vector TY in the z-axis direction according to the projection vector TY and the normal vector FZL of the z-axis to obtain the projection distance zty on the z-axis;

[0125] zty = TY · FZL;

[0126] Step S332: Accumulate the projection distances of the robotic arm segments on the x-axis to obtain the total projection distance zxt on the x-axis; accumulate the projection distances of the robotic arm segments on the y-axis to obtain the total projection distance zyt on the y-axis; accumulate the projection distances of the robotic arm segments on the z-axis to obtain the total projection distance zzt on the z-axis, and combine the grasping vector (Lx, Ly, Lz) to move the robotic arm segments.

[0127] When the projection distance is less than the distance in the corresponding direction of the grasping vector, move in that direction.

[0128] When the projection distance is greater than the distance in the corresponding direction of the grasping vector, move in the opposite direction.

[0129] For example, if the projection distance zxt on the x-axis is less than Lx, move the robotic arm segment in the x-axis direction; if the projection distance zxt on the x-axis is greater than Lx, move the robotic arm segment in the plane formed by the y-axis and the z-axis; similarly, move in the y-axis and z-axis directions.

[0130] When the total projection distance zxt of the robotic arm segment on the x-axis, the total projection distance zyt on the y-axis, and the total projection distance zzt on the z-axis are equal to the values of the grasping vector (Lx, Ly, Lz), obtain the grasping posture of the robotic arm, and grasp the grasping object according to the grasping posture. The specific grasping steps are as follows:

[0131] Please refer to Figure 2 ; Step S3321: Obtain the grasping surface of the grasping object, extract the vertices of the grasping surface, and record the number of vertices as g; obtain the coordinates dz (dzx, dzy, dzz) of the vertices to get dz1, dz2,..., dzg; calculate the central coordinate zxb (ZXx, ZXy, ZXz) of the grasping surface according to the coordinates dz of the vertices:

[0132]

[0133] It should be noted that: the grasping surface of the grasping object refers to the cross-section shown by the grasping object during the grasping process of the robot.

[0134] Step S3322: Connect the central coordinate with any two vertices to obtain vectors xla and xlb; perform a cross-product calculation on vectors xla and xlb to obtain the perpendicular vector zcz (CZx, CZy, CZz) of the grasping surface.

[0135] It should be noted that: when connecting with any two vertices, the vectors obtained after connection should be non-collinear vectors.

[0136] According to the vertical vector zcz (CZx, CZy, CZz) of the grasping surface, calculate its angles θx, θy, θz with the x-axis, y-axis, and z-axis. The specific calculation process is as follows:

[0137]

[0138] Step S3323: Obtain the position of the end effector of the robot, move the position of the end effector of the robot to the center coordinates zxb (ZXx, ZXy, ZXz) of the grasping surface, and adjust the grasping angle of the end effector of the robot so that its angle with the x-axis is θx, the angle with the y-axis is θy, and the angle with the z-axis is θz, and then perform grasping.

[0139] Step S3324: Obtain the adjacent objects of the grasped object and acquire the coordinates Lzb (LZx, LZy, LZz) of the adjacent objects; according to the coordinates Lzb (LZx, LZy, LZz) of the adjacent objects, obtain the coordinates of two vertices of the grasped object that are close to it, connect the vertex coordinates to obtain adjacent sides; map the coordinates Lzb (LZx, LZy, LZz) of the adjacent objects to the adjacent sides to obtain the mapping point YSD (YSx, YSy, YSz), and calculate the distance between the adjacent objects and the adjacent sides of the grasped object according to the mapping point YSD (YSx, YSy, YSz) and the coordinates Lzb (LZx, LZy, LZz) of the adjacent objects to obtain the gap value JXZ;

[0140]

[0141] Please refer to Figure 3 ; according to the coordinates Lzb (LZx, LZy, LZz) of the adjacent objects and the center coordinates zxb (ZXx, ZXy, ZXz) of the grasping surface, obtain the collision vector and calculate the angles of the collision vector to obtain αx, αy, αz;

[0142] Calculate the distance between the coordinates Lzb (LZx, LZy, LZz) of the adjacent objects and the center coordinates zxb (ZXx, ZXy, ZXz) of the grasping surface to obtain the grasping depth zsd;

[0143]

[0144] Obtain the real-time position of the end effector of the robot, and according to the real-time position coordinates of the end effector of the robot, obtain the distance vector Mjl (MLx, Mly, MLz) between the end effector of the robot and the coordinates Lzb (LZx, LZy, LZz) of the adjacent objects; calculate according to the distance vector Mjl to obtain the collision distance pzl;

[0145]

[0146] Obtain the position of the current end effector of the robot, and move it towards the adjacent object according to the collision distance pzl. When the collision distance pzl = 0, if the distance between the position of the end effector of the robot and the adjacent object is greater than the grasping depth zsd, it is judged that grasping can be performed;

[0147] Perform vector summation on the collision vector and the distance vector to obtain the movement vector ydx (YDx, YDy, YDz) of the end effector of the robot. Using the movement vector as the center line, detect the obstacle and obtain the obstacle coordinates; according to the obstacle coordinates and the center coordinates zxb (ZXx, ZXy, ZXz) of the grasping surface, construct obstacle vectors zaw1, zaw2,..., zaws; calculate according to the movement vector ydx (YDx, YDy, YDz) and the obstacle vectors zaw1, zaw2,..., zaws to obtain the change angle of the end effector of the robot;

[0148] Calculate the included angles of the movement vector of the end effector of the robot to obtain the real-time angles βx, βy, βz; obtain the grasping angles θx, θy, θz of the end effector of the robot; according to the real-time angles βx, βy, βz and the grasping angles θx, θy, θz, obtain the movement angle of the end effector of the robot, and compare the movement angle of the end effector of the robot with the change angle of the end effector of the robot. If the movement angle of the end effector of the robot is less than the change angle of the end effector of the robot, it is judged that movement can be performed;

[0149] Obtain the real-time angle of the end effector of the robot, the width mkd and the length mcd of the end effector of the robot; make a judgment in combination with the grasping angles θx, θy, θz of the end effector of the robot; when within the change angle of the end effector of the robot, the angle of the end effector of the robot conforms to the grasping angles θx, θy, θz, and at the same time the width mkd of the end effector of the robot is less than the gap value JXZ, and the length mcd of the end effector of the robot is greater than the grasping depth zsd; it is judged that grasping can be performed.

[0150] Step S4: Obtain the characteristics of the actual grasping object of the robot; combine the grasping object information to perform inspection and judgment on the actual grasping object of the robot;

[0151] Step S41: Extract the image features of the actual grasping object to obtain the image information of each surface of the actual grasping object; according to the image information of each surface of the actual grasping object, extract the image pixels and the image contour, traverse the image pixels of the actual grasping object, obtain the rgb values corresponding to the image pixels, and accumulate the rgb values to obtain the color value of the actual grasping object;

[0152] Obtain the information of the object to be grasped, and calculate the color value of the object to be grasped; compare the color value of the object to be grasped with the color value of the actual grasped object;

[0153] Obtain the image contour of the object to be grasped and the image contour of the actual grasped object, and traverse and compare the image contour of the object to be grasped with the image contour of the actual grasped object;

[0154] According to the comparison result of the color value of the object to be grasped and the color value of the actual grasped object, and the comparison result of the image contours, obtain the inspection result;

[0155] Step S42: According to the inspection result, judge the actual grasped object. If it is judged that the actual grasped object is consistent with the object to be grasped, the grasping behavior is completed; if it is judged that the actual grasped object is inconsistent with the object to be grasped, exclude the actual grasped object, and repeat the grasping behavior according to steps S2 - S3.

[0156] The above formulas are all dimensionless and take their numerical calculations. The formulas are obtained by collecting a large amount of data for software simulation to get a formula closest to the real situation. The preset parameters in the formulas are set by those skilled in the art according to the actual situation. For example, if there are weight coefficients and proportionality coefficients, the sizes of their settings are for quantifying each parameter to obtain a specific value, which is convenient for subsequent comparison. Regarding the sizes of the weight coefficients and proportionality coefficients, as long as they do not affect the proportional relationship between the parameters and the quantified values, it is fine.

[0157] Finally, it should be noted that: the above embodiments are only specific implementation manners of the present invention, used to illustrate the technical solutions of the present invention, rather than limiting it. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: any person skilled in the art within the technical scope disclosed by the present invention can still modify the technical solutions recorded in the foregoing embodiments, or can easily think of changes, or perform equivalent replacements on some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A bionic actuator micro-pickup regulation strategy based on visual field feedback, characterized in that, The control method includes: Step S1: Obtain the robot parameter information and the visual image information; Step S2: According to the visual image information, construct a spatial model; obtain the grasping object information, combine the visual image information to search for the grasping object, obtain the position of the grasping object in the spatial model, and obtain the grasping position; Step S3: Move the robot through the spatial model and the grasping position, obtain the current position of the robot, and obtain the initial position. According to the initial position and the grasping position, obtain the robot grasping path; according to the spatial model, detect the adjacent objects of the grasping object, obtain the gap between the grasping object and the adjacent objects, combine the robot parameter information and the grasping path, calculate the grasping angle during grasping, and according to the grasping angle and the gap between the grasping object and the adjacent objects, detect the robot grasping path. Judge the grasping behavior based on the detection result and control the grasping behavior of the robot; Step S4: Obtain the characteristics of the actual grasping object of the robot; combine the grasping object information to check and judge the actual grasping object of the robot.

2. The macro-picking regulation strategy of a bionic actuator based on field-of-view feedback according to claim 1, wherein The specific steps of step S1 are as follows: Step S11: Obtain the number of joints n of the robot manipulator; measure the length cd of the manipulator arm segments according to the number of joints of the manipulator arm, count the lengths of the manipulator arm segments, and obtain the manipulator arm segment list lbc, lbc = [cd(1), cd(2),..., cd(n - 1)]; Step S12: Obtain the pixel information of the visual image in the space where the robot is located and construct a pixel coordinate system; represent the pixel position in the visual image by (u, v); obtain the depth information of the visual image; the pixel information of the visual image and the depth information of the visual image constitute the visual image information.

3. A macro-picking control strategy for a bionic actuator based on field-of-view feedback according to claim 1, characterized in that, The specific steps of step S2 are as follows: Step S21: Represent the visual image in three dimensions according to the visual image information to obtain the three-dimensional data of the visual image, and construct a spatial model according to the three-dimensional data of the visual image; Step S22: Obtain the grasping object information; compare the visual image according to the grasping object information to obtain the image position of the grasping object, and obtain the spatial position of the grasping object in the spatial model according to the image position of the grasping object to obtain the grasping position (Zx, Zy, Zz).

4. A micro-picking control strategy for a bionic actuator based on field-of-view feedback according to claim 3, characterized in that, The specific steps of step S21 are as follows: Step S211: Obtain the current position as the origin position (0, 0, 0), define the horizontal right direction as the x-axis, define the horizontal downward direction as the y-axis; define the direction along the optical axis outward as the z-axis; construct a three-dimensional coordinate system according to the origin position, the x-axis, the y-axis, and the z-axis; Step S212: Calculate the three-dimensional coordinates of the image pixels according to the visual image information and the three-dimensional coordinate system, and construct a spatial model from the three-dimensional coordinates of the image pixels; The specific steps of step S212 are as follows: Obtain the depth information of the visual image from the visual image information, and obtain the depth corresponding to the image pixel according to the depth information of the visual image to obtain the coordinate Sz corresponding to the image pixel in the z-axis direction; Obtain the optical center coordinates (Cx, Cy) based on the intersection position of the pixel coordinate system and the optical axis; obtain the focal lengths (Fx, Fy). Obtain the position u corresponding to the image pixel in the horizontal direction of the pixel coordinate system and the position v corresponding to the image pixel in the vertical direction of the pixel coordinate system. Combine the optical center coordinates (Cx, Cy), the focal lengths (Fx, Fy), and the coordinate Sz corresponding to the image pixel on the z-axis to calculate the coordinate Sx corresponding to the image pixel in the x-axis direction and the coordinate Sy corresponding to the image pixel in the y-axis direction. Obtain the three-dimensional coordinates (Sx, Sy, Sz) of the image pixel based on the coordinate Sz corresponding to the image pixel in the z-axis direction, the coordinate Sx corresponding to the image pixel in the x-axis direction, and the coordinate Sy corresponding to the image pixel in the y-axis direction; fill the space according to the three-dimensional coordinates of the image pixel to obtain a space model. Obtain the point cloud data of the object in the space. According to the point cloud data, obtain the object point coordinates (dbx, dby, dbz) in the point cloud data and the reflected light intensity corresponding to the point coordinates. According to the object point coordinates (dbx, dby, dbz) in the point cloud data, obtain the reflected light intensity of the corresponding point coordinates in the space model. Compare the reflected light intensity corresponding to the point coordinates in the point cloud data with the reflected light intensity of the corresponding point coordinates in the space model. If the two reflected light intensities are different, replace the object in the space model according to the object point coordinates of the point cloud data and optimize the space model.

5. A macro-picking control strategy for a bionic actuator based on field-of-view feedback according to claim 1, characterized in that, The specific steps of step S3 are as follows: Step S31: Move the robot through the space model and the grasping position, move the robot towards the grasping position. When the robot cannot move, obtain the three-dimensional coordinates of the current position of the robot to get the initial position (Jx, Jy, Jz); connect the initial position and the grasping position to obtain the grasping vector (Lx, Ly, Lz), and use the grasping vector as the grasping path of the robot. Step S32: According to the robot parameter information, obtain the list of manipulator arm segments lbc, lbc = [cd(1), cd(2),..., cd(n - 1)]; combine the grasping path of the robot to judge the grasping behavior. Step S33: Fit the manipulator arm segments to the grasping path of the robot to obtain the grasping posture of the manipulator arm segments, and grasp the grasping object according to the grasping posture.

6. A macro-picking control strategy for a bionic actuator based on field-of-view feedback according to claim 5, characterized in that The specific steps of step S32 are as follows: Step S321: Calculate the maximum reachable distance of the manipulator arm according to the list of manipulator arm segments lbc = [cd(1), cd(2),..., cd(n - 1)] to obtain the maximum distance ZCJ: ZCJ = cd(1) + cd(2) +...... + cd(n - 1); Step S322: According to the grasping path of the robot, obtain the grasping vector (Lx, Ly, Lz), and calculate the distance between the robot and the grasping object according to the grasping vector to obtain the grasping distance ZQJ: Step S323: Judge whether the robot can grasp according to the maximum distance ZCJ and the grasping distance ZQJ: If ZCJ ≥ ZQJ, it indicates that the robot can grasp the grasping object; enter step S33; If ZCJ < ZQJ, it indicates that the robot cannot grasp the object to be grasped, and an alarm error is given.

7. A macro-pickup control strategy for a bionic actuator based on field-of-view feedback according to claim 5, characterized in that The specific steps of step S33 are as follows: Step S331: Obtain the real-time position of the robotic arm segment, and calculate the projections of the robotic arm segment on the three-dimensional coordinate system according to the real-time position of the robotic arm segment. The specific calculation is as follows: Obtain the joint coordinates ZB1(Z1x, Z1y, Z1z) and ZB2(Z2x, Z2y, Z2z) adjacent to the robotic arm segment, and calculate the projection vector TY(Z1x - Z2x, Z1y - Z2y, Z1z - Z2z) according to the joint coordinates; Obtain the normal vector FXL(XAx, XAy, XAz) in the x-axis direction, and calculate the projection of the projection vector TY in the x-axis direction according to the projection vector TY and the normal vector FXL of the x-axis to obtain the projection distance xty on the x-axis; Similarly, obtain the projection distance yty on the y-axis and the projection distance zty on the z-axis; Step S332: Accumulate the projection distances of the robotic arm segment on the x-axis to obtain the total projection distance zxt on the x-axis; accumulate the projection distances of the robotic arm segment on the y-axis to obtain the total projection distance zyt on the y-axis; accumulate the projection distances of the robotic arm segment on the z-axis to obtain the total projection distance zzt on the z-axis. Combine the grasping vector (Lx, Ly, Lz) to move the robotic arm segment; When the projection distance is less than the distance in the corresponding direction of the grasping vector, move in that direction; When the projection distance is greater than the distance in the corresponding direction of the grasping vector, move in the opposite direction; When the total projection distances zxt, zyt, zzt of the robotic arm segment on the x-axis, y-axis, and z-axis are equal to the values of the grasping vector (Lx, Ly, Lz), obtain the grasping posture of the robotic arm, and grasp the object to be grasped according to the grasping posture.

8. A macro-picking regulation strategy for a bionic actuator based on field-of-view feedback according to claim 7, characterized in that The specific steps of step S332 are as follows: Obtain the grasping surface of the object to be grasped, extract the vertices of the grasping surface, and record the number of vertices as g; obtain the coordinates dz(dzx, dzy, dzz) of the vertices to get dz1, dz2,..., dzg; calculate the central coordinate zxb(ZXx, ZXy, ZXz) of the grasping surface according to the coordinates dz of the vertices: Connect the central coordinate with any two vertices to obtain vectors xla and xlb; perform a cross product calculation on vectors xla and xlb to obtain the perpendicular vector zcz(CZx, CZy, CZz) of the grasping surface; According to the perpendicular vector zcz(CZx, CZy, CZz) of the grasping surface, calculate its angles θx, θy, θz with the x-axis, y-axis, and z-axis; Obtain the position of the end effector of the robot, move the position of the end effector of the robot to the central coordinate of the grasping surface, adjust the angle of the end effector of the robot so that its angle with the x-axis is θx, its angle with the y-axis is θy, and its angle with the z-axis is θz, and perform grasping. Obtain the adjacent objects of the grasping object, and obtain the coordinates Lzb (LZx, LZy, LZz) of the adjacent objects; according to the coordinates of the adjacent objects, obtain the coordinates of two vertices of the grasping object that are close to it, connect the vertex coordinates to obtain adjacent sides; map the coordinates Lzb (LZx, LZy, LZz) of the adjacent objects to the adjacent sides to obtain the mapping points YSD (YSx, YSy, YSz), and calculate the distance between the adjacent sides of the adjacent object and the grasping object according to the mapping points and the coordinates of the adjacent objects to obtain the clearance value JXZ; According to the coordinates of the adjacent object and the central coordinates of the grasping surface, obtain the collision vector, and calculate the included angle of the collision vector to obtain αx, αy, αz.

9. A macro-picking regulation strategy for a bionic actuator based on field-of-view feedback according to claim 8, characterized in that The step S332 further includes: Calculate the distance between the coordinates of the adjacent object and the central coordinates of the grasping surface to obtain the grasping depth zsd; Obtain the distance vector Mjl (MLx, Mly, MLz) between the end effector of the robot and the coordinates Lzb (LZx, LZy, LZz) of the adjacent object; calculate according to the distance vector Mjl to obtain the collision distance pzl; Obtain the current position of the end effector of the robot, and move it towards the adjacent object according to the collision distance pzl. When the collision distance pzl = 0, if the distance between the position of the end effector of the robot and the adjacent object is greater than the grasping depth zsd, it is determined that grasping can be performed.

10. A macro-picking regulation strategy for a bionic actuator based on visual field feedback according to claim 9, characterized in that, The step S332 further includes: Perform vector summation on the collision vector and the distance vector to obtain the movement vector ydx (YDx, YDy, YDz) of the end effector of the robot. Take the movement vector as the center line to detect obstacles and obtain obstacle coordinates; according to the obstacle coordinates and the central coordinates zxb (ZXx, ZXy, ZXz) of the grasping surface, construct obstacle vectors zaw1, zaw2,..., zaws; calculate according to the movement vector ydx (YDx, YDy, YDz) and the obstacle vectors zaw1 to zaws to obtain the change angle of the end effector of the robot; Calculate the included angle of the movement vector of the end effector of the robot to obtain the real-time angles βx, βy, βz; obtain the grasping angles θx, θy, θz of the end effector of the robot; according to the real-time angles βx, βy, βz and the grasping angles θx, θy, θz, obtain the movement angle of the end effector of the robot, and compare the movement angle of the end effector of the robot with the change angle of the end effector of the robot. If the movement angle of the end effector of the robot is less than the change angle of the end effector of the robot, it is determined that movement can be performed; Obtain the real-time angle of the end effector of the robot, the width mkd and the length mcd of the end effector of the robot; make a judgment in combination with the grasping angles θx, θy, θz of the end effector of the robot; when within the change angle of the end effector of the robot, the angle of the end effector of the robot conforms to the grasping angles θx, θy, θz, and at the same time the width mkd of the end effector of the robot is less than the clearance value JXZ, and the length mcd of the end effector of the robot is greater than the grasping depth zsd; it is determined that grasping can be performed.

Images