Bionical executor micro-distance pickup regulation strategy based on visual field feedback

Abstract

The application provides a kind of micro-distance pickup regulation strategy based on visual field feedback bionic actuator, belongs to robot control field;Solve the problem of low robot grasping precision;Specific as follows:Step S1: obtain robot parameter information and visual image information;Step S2: construct space model;According to the visual image information, the grasping object is searched, the position of the grasping object in the space model is obtained, and the grasping position is obtained;Step S3: move the robot, get the initial position, according to the initial position and the grasping position, get the robot grasping path;According to the robot grasping path and the robot parameter information, the grasping behavior of the robot is controlled;Step S4: by grasping object information, the robot grasping object is inspected and judged;The application controls the robot, accurately grasps the object by image processing, inspects the grasping object, and improves the robot grasping accuracy.

Description

TECHNICAL FIELD

[0001] The application discloses a bionic actuator micro-distance pickup regulation strategy based on visual field feedback and relates to the field of robot control. BACKGROUND

[0002] The existing bionic actuator micro-distance pickup regulation strategy based on visual field feedback has the following disadvantages:

[0003] Limited application range: the existing robot precise grasping control method is mainly used for grasping control of specific objects by a mechanical arm in a fixed range, the grasping range is small, and the grasping objects are limited; and the generalization ability for complex scenes is insufficient;

[0004] Limitations of grasping strategies: traditional methods rely on pre-defined grasping templates (such as fixed angles of parallel grippers and fixed adsorption surfaces of suction cups), and it is difficult to adapt to shape, material or posture changes of target objects; for example, when grasping an object placed obliquely, the fixed grasping angle cannot be fitted to the target surface;

[0005] Insufficient coordination of mechanical arm segments: in the movement process of a multi-joint mechanical arm, dynamic synchronization cannot be achieved between the segments (joints), resulting in a decrease in the movement precision of an end effector (such as a gripper or a tool) and a decrease in grasping precision;

[0006] Incomplete consideration of special cases: a three-dimensional coordinate acquisition method is disclosed in the patent application file with the publication number CN112372641A, which acquires object coordinates through internal parameter calibration and hand-eye calibration of a global camera and a mechanical arm end camera; in some special cases (such as excessively low and high brightness), the coordinate acquisition by the camera is prone to cause parameter loss, and thus errors are generated, resulting in low precision of a three-dimensional model. SUMMARY

[0007] In view of the disadvantages of the prior art, the application aims to provide a bionic actuator micro-distance pickup regulation strategy based on visual field feedback, and aims to solve the problem of robot control.

[0008] In order to achieve the above-mentioned purpose, the application is implemented by the following technical solutions:

[0009] A bionic actuator micro-distance pickup regulation strategy based on visual field feedback comprises the following steps:

[0010] Step S1: acquiring robot parameter information and visual image information;

[0011] Step S2: constructing a space model according to the visual image information; acquiring grasping object information, searching for the grasping object in combination with the visual image information, acquiring the position of the grasping object in the space model, and obtaining a grasping position;

[0012] Step S3: moving the robot according to the space model and the grabbing position, obtaining the current position of the robot, obtaining the initial position, obtaining the robot grabbing path according to the initial position and the grabbing position, detecting the adjacent objects of the grabbing object according to the space model, obtaining the gap between the grabbing object and the adjacent objects, calculating the grabbing angle during grabbing in combination with the robot parameter information and the grabbing path, detecting the robot grabbing path according to the grabbing angle and the gap between the grabbing object and the adjacent objects, judging the grabbing behavior according to the detection result, and controlling the grabbing behavior of the robot;

[0013] Step S4: obtaining the characteristics of the actual grabbing object of the robot; and verifying and judging the actual grabbing object of the robot in combination with the grabbing object information.

[0014] Further, the specific steps of step S1 are as follows:

[0015] Step S11: obtaining the number n of joints of the robot arm; measuring the arm segment length cd of the robot arm according to the number of joints of the robot arm, counting the arm segment length of the robot arm, obtaining the arm segment list lbc, lbc = [cd(1), cd(2), …, cd(n-1)];

[0016] Step S12: obtaining the pixel information of the visual image of the space where the robot is located, setting the upper left corner of the visual image as the image origin, setting the horizontal right direction as the horizontal axis, and setting the vertical downward direction as the vertical axis to construct a pixel coordinate system; representing the pixel position in the visual image as (u, v); wherein (u, v) represents the pixel point at the u-th row and the v-th column from the image origin; obtaining the depth information of the visual image; and the pixel information of the visual image and the depth information of the visual image constitute the visual image information.

[0017] Further, the specific steps of step S2 are as follows:

[0018] Step S21: representing the visual image in three dimensions according to the visual image information to obtain three-dimensional data of the visual image, and constructing a space model according to the three-dimensional data of the visual image;

[0019] Step S22: obtaining the grabbing object information; comparing the visual image according to the grabbing object information to obtain the image position of the grabbing object, obtaining the space position of the grabbing object in the space model according to the image position of the grabbing object, and obtaining the grabbing position (Zx, Zy, Zz).

[0020] Further, the specific steps of step S21 are as follows:

[0021] Step S211: Obtain the current position as the origin position (0, 0, 0), define the current view angle horizontal right direction as the x-axis, define the current view angle horizontal down direction as the y-axis, and define the outward direction along the optical axis as the z-axis; construct a three-dimensional coordinate system according to the origin position, the x-axis, the y-axis, and the z-axis;

[0022] Step S212: According to the visual image information and the three-dimensional coordinate system, calculate the three-dimensional coordinates of the image pixels, and construct a space model according to the three-dimensional coordinates of the image pixels.

[0023] Further, the specific steps of step S212 are as follows:

[0024] According to the visual image information, obtain the depth information of the visual image, obtain the depth corresponding to the image pixel according to the depth information of the visual image, and obtain the coordinate Sz corresponding to the image pixel in the z-axis direction;

[0025] According to the intersection position of the pixel coordinate system and the optical axis, obtain the optical center coordinates (Cx, Cy); obtain the focal length (Fx, Fy);

[0026] According to the pixel information of the visual image, obtain the position u of the image pixel in the horizontal direction of the pixel coordinate system, and calculate the coordinate Sx corresponding to the image pixel in the x-axis direction by combining the horizontal coordinate Cx of the optical center coordinates, the horizontal coordinate Fx of the focal length, and the coordinate Sz corresponding to the image pixel in the z-axis;

[0027] According to the pixel information of the visual image, obtain the position v of the image pixel in the vertical direction of the pixel coordinate system, and calculate the coordinate Sy corresponding to the image pixel in the y-axis direction by combining the vertical coordinate Cy of the optical center coordinates, the vertical coordinate Fy of the focal length, and the coordinate Sz corresponding to the image pixel in the z-axis;

[0028] According to the coordinate Sz corresponding to the image pixel in the z-axis direction, the coordinate Sx corresponding to the image pixel in the x-axis direction, and the coordinate Sy corresponding to the image pixel in the y-axis direction, obtain the three-dimensional coordinates (Sx, Sy, Sz) of the image pixel; fill the space according to the three-dimensional coordinates of the image pixel, and obtain the space model;

[0029] Obtain the point cloud data of the object in the space, obtain the reflection light intensity corresponding to the object point coordinates (dbx, dby, dbz) in the point cloud data according to the point cloud data, obtain the reflection light intensity of the corresponding point coordinates in the space model according to the object point coordinates (dbx, dby, dbz) in the point cloud data, compare the reflection light intensity corresponding to the point coordinates in the point cloud data with the reflection light intensity of the corresponding point coordinates in the space model, and if the reflection light intensities are different, replace the object in the space model according to the object point coordinates of the point cloud data, and optimize the space model.

[0030] Further, the specific steps of step S3 are as follows:

[0031] Step S31: moving the robot through the space model and the grabbing position, moving the robot to the grabbing position, when the robot cannot move, obtaining the three-dimensional coordinates of the current position of the robot to obtain the initial position (Jx, Jy, Jz); according to the initial position and the grabbing position, taking the initial position as the starting position and the grabbing position as the target position, connecting the starting position and the target position to obtain the grabbing vector (Lx, Ly, Lz), the grabbing vector (Lx, Ly, Lz) = (Zx, Zy, Zz) - (Jx, Jy, Jz), taking the grabbing vector as the robot grabbing path;

[0032] Step S32: according to the robot parameter information, obtaining the mechanical arm arm segment list lbc, lbc = [cd(1), cd(2), …, cd(n-1)]; in combination with the robot grabbing path, judging the grabbing behavior;

[0033] Step S33: fitting the mechanical arm arm segment to the robot grabbing path to obtain the grabbing posture of the mechanical arm arm segment, and grabbing the grabbing object according to the grabbing posture.

[0034] Further, the specific steps of step S32 are as follows:

[0035] Step S321: according to the mechanical arm arm segment list lbc = [cd(1), cd(2), …, cd(n-1)], calculating the longest grabbable distance of the mechanical arm to obtain the longest distance ZCJ:

[0036] ZCJ = cd(1) + cd(2) + … + cd(n-1);

[0037] Step S322: according to the robot grabbing path, obtaining the grabbing vector (Lx, Ly, Lz), calculating the distance between the robot and the grabbing object according to the grabbing vector to obtain the grabbing distance ZQJ:

[0038]

[0039] Step S323: according to the longest distance ZCJ and the grabbing distance ZQJ, judging whether the robot can grab:

[0040] If ZCJ ≥ ZQJ, it indicates that the robot can grab the grabbing object; enter step S33;

[0041] If ZCJ < ZQJ, it indicates that the robot cannot grab the grabbing object, and an alarm error is performed.

[0042] Further, the specific steps of the step S33 are as follows:

[0043] Step S331: Real-time positions of the arm segments of the robot arm are acquired, and projections of the arm segments of the robot arm on a three-dimensional coordinate system are calculated according to the real-time positions of the arm segments of the robot arm, and the calculation is as follows:

[0044] Adjacent joint coordinates ZB1(Z1x, Z1y, Z1z) and ZB2(Z2x, Z2y, Z2z) of the arm segments of the robot arm are acquired, and a projection vector TY(Z1x-Z2x, Z1y-Z2y, Z1z-Z2z) is calculated according to the joint coordinates ZB1(Z1x, Z1y, Z1z) and ZB2(Z2x, Z2y, Z2z).

[0045] A normal vector FXL(XAx, XAy, XAz) in the x-axis direction is acquired, and a projection distance xty of the projection vector TY on the x-axis is calculated according to the projection vector TY and the normal vector FXL of the x-axis, so as to obtain the projection distance xty on the x-axis.

[0046] A normal vector FYL(YAx, YAy, YAz) in the y-axis direction is acquired, and a projection distance yty of the projection vector TY on the y-axis is calculated according to the projection vector TY and the normal vector FYL of the y-axis, so as to obtain the projection distance yty on the y-axis.

[0047] A normal vector FZL(ZAx, ZAy, ZAz) in the z-axis direction is acquired, and a projection distance zty of the projection vector TY on the z-axis is calculated according to the projection vector TY and the normal vector FZL of the z-axis, so as to obtain the projection distance zty on the z-axis.

[0048] Step S332: The projection distances of the arm segments of the robot arm on the x-axis are accumulated to obtain a total projection distance zxt on the x-axis, the projection distances of the arm segments of the robot arm on the y-axis are accumulated to obtain a total projection distance zyt on the y-axis, and the projection distances of the arm segments of the robot arm on the z-axis are accumulated to obtain a total projection distance zzt on the z-axis, and the arm segments of the robot arm are moved in combination with a grasping vector (Lx, Ly, Lz).

[0049] When the projection distance is less than the distance in the direction corresponding to the grasping vector, the robot arm is moved in the direction.

[0050] When the projection distance is greater than the distance in the direction corresponding to the grasping vector, the robot arm is moved in the opposite direction.

[0051] When the total projection distances zxt, zyt and zzt of the arm segments of the robot arm on the x-axis, the y-axis and the z-axis are equal to the values of the grasping vector (Lx, Ly, Lz), a grasping posture of the robot arm is obtained, and the grasping object is grasped according to the grasping posture.

[0052] Further, the specific steps of the step S332 are as follows:

[0053] The grabbing surface of the grabbing object is obtained, the vertexes of the grabbing surface are extracted, and the number of vertexes is recorded as g; the coordinates dz (dzx, dzy, dzz) of the vertexes are obtained, and dz1, dz2, …, dzg are obtained; the center coordinate zxb (ZXx, ZXy, ZXz) of the grabbing surface is calculated according to the coordinates dz of the vertexes:

[0054]

[0055] The center coordinate is connected with any two vertexes to obtain vectors xla and xlb; the cross product of the vectors xla and xlb is calculated to obtain the normal vector zcz (CZx, CZy, CZz) of the grabbing surface;

[0056] According to the normal vector zcz (CZx, CZy, CZz) of the grabbing surface, the included angles θx, θy and θz of the normal vector zcz (CZx, CZy, CZz) with the x-axis, the y-axis and the z-axis are calculated;

[0057] The position of the robot end effector is obtained, the position of the robot end effector is moved to the center coordinate zxb (ZXx, ZXy, ZXz) of the grabbing surface, the angle of the robot end effector is adjusted, the included angle of the robot end effector with the x-axis is θx, the included angle of the robot end effector with the y-axis is θy, and the included angle of the robot end effector with the z-axis is θz, and the grabbing is performed;

[0058] The adjacent object of the grabbing object is obtained, the coordinates Lzb (LZx, LZy, LZz) of the adjacent object are obtained; according to the coordinates Lzb (LZx, LZy, LZz) of the adjacent object, the two vertex coordinates adjacent to the grabbing object are obtained, the vertex coordinates are connected to obtain the adjacent edge; the coordinates Lzb (LZx, LZy, LZz) of the adjacent object are mapped to the adjacent edge to obtain the mapping point YSD (YSx, YSy, YSz), and according to the mapping point YSD (YSx, YSy, YSz) and the coordinates Lzb (LZx, LZy, LZz) of the adjacent object, the distance between the adjacent object and the adjacent edge of the grabbing object is calculated to obtain the gap value JXZ;

[0059]

[0060] According to the coordinates Lzb (LZx, LZy, LZz) of the adjacent object and the center coordinate zxb (ZXx, ZXy, ZXz) of the grabbing surface, the collision vector is obtained, the included angle of the collision vector is calculated, and αx, αy and αz are obtained;

[0061] A distance between the coordinates Lzb (LZx, LZy, LZz) of the adjacent object and the center coordinates zxb (ZXx, ZXy, ZXz) of the grabbing surface is calculated to obtain a grabbing depth zsd;

[0062]

[0063] An instantaneous position of the robot end effector is obtained, and a distance vector Mjl (MLx, MLy, MLz) between the instantaneous position coordinates of the robot end effector and the coordinates Lzb (LZx, LZy, LZz) of the adjacent object is obtained; a collision distance pzl is calculated according to the distance vector Mjl.

[0064]

[0065] The position of the current robot end effector is obtained, and the adjacent object is moved according to the collision distance pzl; when the collision distance pzl = 0, if the distance between the position of the robot end effector and the adjacent object is greater than the grabbing depth zsd, it is determined that the grabbing can be performed.

[0066] Further, the step S332 further includes:

[0067] The collision vector and the distance vector are vector summed to obtain a movement vector ydx (YDx, YDy, YDz) of the robot end effector, the movement vector is taken as a center line, the obstacle is detected, and obstacle coordinates are obtained; obstacle vectors zaw1, zaw2, …, zaws are constructed according to the obstacle coordinates and the center coordinates zxb (ZXx, ZXy, ZXz) of the grabbing surface; and a change angle of the robot end effector is calculated according to the movement vector ydx (YDx, YDy, YDz) and the obstacle vectors zaw1, zaw2, …, zaws.

[0068] The angle of the movement vector of the robot end effector is calculated to obtain real-time angles beta x, beta y and beta z, the grabbing angles theta x, theta y and theta z of the robot end effector are obtained, the movement angle of the robot end effector is obtained according to the real-time angles beta x, beta y and beta z and the grabbing angles theta x, theta y and theta z, the movement angle of the robot end effector is compared with the change angle of the robot end effector, if the movement angle of the robot end effector is smaller than the change angle of the robot end effector, it is judged that movement can be performed, the real-time angles of the robot end effector, the width mkd and the length mcd of the robot end effector are obtained, and the grabbing angles theta x, theta y and theta z of the robot end effector are combined for judgment, when the angles of the robot end effector are within the change angles b hx, b hy and b hz, the angles of the robot end effector meet the grabbing angles theta x, theta y and theta z, the width mkd of the robot end effector is smaller than the gap value JXZ, and the length mcd of the robot end effector is greater than the grabbing depth z sd, it is judged that grabbing can be performed.

[0069] Compared with the prior art, the beneficial effects of the present application are:

[0070] Extended application range: a three-dimensional space model is constructed for a grabbing range space, objects in the space range are grabbed according to a grabbing object, dynamic large-range grabbing is realized, and the robot grabbing is generalized for use in multiple fields;

[0071] Optimized grabbing strategy: the grabbing object is specifically analyzed, the mechanical arm is adjusted according to the center position and the offset angle of the grabbing object, the mechanical arm grabbing can adapt to different shapes or attitude changes, and the grabbing stability and applicability are enhanced;

[0072] Synchronous adjustment of the mechanical arm segments: the mechanical arm segments are divided into separate individuals for motion control, the original motion trajectory of the mechanical arm is refined, the grabbing speed is optimized, the movement trajectory of the mechanical arm is decomposed, the movement trajectory of each mechanical arm segment is determined, and the specific grabbing path is determined;

[0073] Optimized model construction: the patent application file with publication number CN112372641A discloses a three-dimensional coordinate acquisition method, based on which, the present application introduces a laser radar to scan objects in the space to obtain point cloud data of the objects in the space, combines the point cloud data with the original model, and prevents special cases of excessively low and excessively high brightness; the model construction is optimized to make the model more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0074] Other features, objects and advantages of the present application will become more apparent through reading the following detailed description of the non-limiting embodiments with reference to the accompanying drawings:

[0075] Figure 1A schematic diagram of the method of the present application;

[0076] Figure 2 A schematic diagram of the main data processing of the present application;

[0077] Figure 3 A schematic diagram of the present application grabbing; DETAILED DESCRIPTION

[0078] In order to make the above objectives, characteristics and advantages of the present application more apparent, obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0079] Example One

[0080] Please refer to Figure 1 A micro-distance pickup regulation strategy of a bionic actuator based on visual field feedback includes:

[0081] It should be noted that the bionic actuator refers to an intelligent device that simulates the motion mechanism or sensing function of a living being, which is a robot in the present application.

[0082] Step S1: Obtain robot parameter information and visual image information;

[0083] Step S11: Obtain the number of joints n of the robot mechanical arm; measure the mechanical arm segment length cd according to the number of joints of the mechanical arm, and statistically obtain the mechanical arm segment list lbc, lbc=[cd(1), cd(2), …, cd(n-1)];

[0084] It should be noted that the mechanical arm segment refers to a rigid structure part connecting two adjacent joints in the mechanical arm, which is responsible for transmitting motion and force; the number thereof is the number of joints minus 1; cd(1) represents the length of the first mechanical arm segment.

[0085] Step S12: Through the depth camera, the pixel information of the visual image of the space where the robot is located is obtained, the upper left corner of the visual image is set as the image origin, the horizontal right direction is set as the horizontal axis, and the vertical downward direction is set as the vertical axis to construct the pixel coordinate system; the pixel position in the visual image is represented by (u, v); wherein (u, v) represents the pixel point at the u row and the v column from the image origin; the depth camera obtains the depth information of the visual image; the pixel information of the visual image and the depth information of the visual image constitute the visual image information;

[0086] Depth camera is a device that can obtain the depth information of objects in a scene by measuring the distance between the object and the camera, and construct a three-dimensional space model. Compared with traditional two-dimensional cameras, depth cameras can provide more rich spatial information, and are widely used in three-dimensional reconstruction, human-computer interaction, robot navigation, augmented reality (AR), virtual reality (VR), automatic driving and other fields.

[0087] Step S2: constructing a space model according to the visual image information; obtaining the information of the grasped object, searching for the grasped object in combination with the visual image information, obtaining the position of the grasped object in the space model, and obtaining the grasping position;

[0088] Step S21: representing the visual image in three dimensions according to the visual image information to obtain three-dimensional data of the visual image, and constructing a space model according to the three-dimensional data of the visual image; the specific construction process is as follows:

[0089] Step S211: taking the position of the depth camera as the origin position (0, 0, 0), defining the horizontal right direction of the camera view as the x-axis, and defining the horizontal downward direction of the camera view as the y-axis; defining the outward direction along the camera optical axis as the z-axis; constructing a three-dimensional coordinate system according to the origin position, the x-axis, the y-axis and the z-axis;

[0090] Step S212: calculating the three-dimensional coordinates of the image pixels according to the visual image information and the three-dimensional coordinate system, and constructing a space model from the three-dimensional coordinates of the image pixels.

[0091] Step S2121: obtaining the depth information of the visual image from the visual image information, obtaining the depth corresponding to the image pixels according to the depth information of the visual image, and obtaining the coordinate Sz corresponding to the image pixels in the z-axis direction;

[0092] Step S2122: obtaining the optical center coordinates (Cx, Cy) according to the intersection position of the pixel coordinate system and the camera optical axis; and obtaining the focal length (Fx, Fy) according to the depth camera;

[0093] It should be noted that the focal length represents the ability of the camera lens to focus light rays in three-dimensional space onto the image plane, measured in pixels.

[0094] According to the pixel information of the visual image, the position u corresponding to the image pixels in the horizontal direction of the pixel coordinate system is obtained, and the coordinate Cx of the optical center in the horizontal direction, the coordinate Fx of the focal length in the horizontal direction, and the coordinate Sz of the image pixels in the z-axis are combined to calculate the coordinate Sx corresponding to the image pixels in the x-axis direction;

[0095]

[0096] According to the pixel information of the visual image, a position v corresponding to the image pixel in a vertical direction of a pixel coordinate system is obtained, and a coordinate Sy corresponding to the image pixel in a y-axis direction is calculated by combining a coordinate Cy of the optical center in the vertical direction, a coordinate Fy of the focal length in the vertical direction, and a coordinate Sz corresponding to the image pixel in a z-axis direction.

[0097]

[0098] Step S2123: obtaining a three-dimensional coordinate (Sx, Sy, Sz) of the image pixel according to the coordinate Sz corresponding to the image pixel in the z-axis direction, the coordinate Sx corresponding to the image pixel in the x-axis direction, and the coordinate Sy corresponding to the image pixel in the y-axis direction; and filling the space according to the three-dimensional coordinate of the image pixel to obtain a space model.

[0099] Step S2124: scanning the objects in the space by the laser radar to obtain point cloud data of the objects in the space, and verifying the space model according to the point cloud data, specifically as follows:

[0100] According to the point cloud data, the reflection light intensity corresponding to the object point coordinates (dbx, dby, dbz) in the point cloud data and the point coordinates is obtained, the reflection light intensity of the corresponding point coordinates in the space model is obtained according to the object point coordinates (dbx, dby, dbz) in the point cloud data, the reflection light intensity corresponding to the point coordinates in the point cloud data is compared with the reflection light intensity of the corresponding point coordinates in the space model, if the reflection light intensities are different, the object in the space model is replaced according to the object point coordinates of the point cloud data, and the space model is optimized.

[0101] It should be noted that the reflection light intensity of the laser radar is more accurate, less affected by environmental light, and suitable for long-distance detection (up to hundreds of meters). Its high resolution and anti-interference ability make it perform excellently in complex environments.

[0102] The optimized space model and the point cloud data are optimized by a feature matching algorithm to optimize point cloud registration, eliminate spatial offset between devices, fill the hollow area caused by occlusion or reflection of the depth camera according to the laser radar point cloud, constrain the noise points of the laser radar point cloud according to the dense data of the depth camera, and improve the accuracy of the model.

[0103] Step S22: obtaining the information of the grasped object; comparing the visual image according to the information of the grasped object to obtain the image position of the grasped object, and obtaining the spatial position of the grasped object in the space model according to the image position of the grasped object to obtain the grasping position (Zx, Zy, Zz).

[0104] It should be noted that the information of the grasped object refers to the color and shape of the grasped object.

[0105] Step S3: moving the robot according to the space model and the grabbing position, obtaining the current position of the robot to obtain an initial position, obtaining a robot grabbing path according to the initial position and the grabbing position, detecting the adjacent objects of the grabbing object according to the space model, obtaining the gap between the grabbing object and the adjacent objects, calculating the grabbing angle during grabbing in combination with the robot parameter information and the grabbing path, detecting the robot grabbing path according to the grabbing angle and the gap between the grabbing object and the adjacent objects, judging the grabbing behavior according to the detection result, and controlling the grabbing behavior of the robot;

[0106] Step S31: moving the robot according to the space model and the grabbing position, moving the robot to the grabbing position, obtaining the three-dimensional coordinates of the current position of the robot when the robot cannot move to obtain an initial position (Jx, Jy, Jz), connecting the initial position and the grabbing position as a starting position and a target position to obtain a grabbing vector (Lx, Ly, Lz), and taking the grabbing vector as the robot grabbing path;

[0107] It should be noted that the grabbing vector is connected with the initial position and the grabbing position, and the grabbing vector represents the shortest distance between the initial position and the grabbing position.

[0108] Step S32: obtaining a mechanical arm arm segment list lbc according to the robot parameter information, lbc = [cd(1), cd(2), …, cd(n-1)], and judging the grabbing behavior in combination with the robot grabbing path.

[0109] Step S321: calculating the longest grabbable distance of the mechanical arm according to the mechanical arm arm segment list lbc = [cd(1), cd(2), …, cd(n-1)] to obtain a longest distance ZCJ.

[0110] ZCJ = cd(1) + cd(2) + … + cd(n-1).

[0111] It should be noted that when all the mechanical arm arm segments are in the same straight line direction, the grabbable distance of the mechanical arm is the longest.

[0112] Step S322: obtaining the grabbing vector (Lx, Ly, Lz) according to the robot grabbing path, calculating the distance between the robot and the grabbing object according to the grabbing vector to obtain a grabbing distance ZQJ.

[0113]

[0114] Step S323: According to the longest distance ZCJ and the grabbing distance ZQJ, whether the robot can grab is judged:

[0115] If ZCJ≥ZQJ, it indicates that the robot can grab the grabbing object; step S33 is entered;

[0116] If ZCJ<ZQJ, it indicates that the robot cannot grab the grabbing object, and an alarm error is performed.

[0117] Step S33: The robot grabbing path is fitted by the arm segment, the grabbing posture of the arm segment is obtained, and the grabbing object is grabbed according to the grabbing posture;

[0118] Step S331: The real-time position of the arm segment is obtained, the projection of the arm segment on the three-dimensional coordinate system is calculated according to the real-time position of the arm segment, and the specific calculation is as follows:

[0119] The adjacent joint coordinates ZB1(Z1x, Z1y, Z1z) and ZB2(Z2x, Z2y, Z2z) of the arm segment are obtained, the projection vector TY(Z1x-Z2x, Z1y-Z2y, Z1z-Z2z) is calculated according to the joint coordinates ZB1(Z1x, Z1y, Z1z) and ZB2(Z2x, Z2y, Z2z);

[0120] The normal vector FXL(XAx, XAy, XAz) in the x-axis direction is obtained, the projection of the projection vector TY in the x-axis direction is calculated according to the projection vector TY and the normal vector FXL of the x-axis, and the projection distance xty in the x-axis direction is obtained.

[0121] xty=TY·FXL;

[0122] The normal vector FYL(YAx, YAy, YAz) in the y-axis direction is obtained, the projection of the projection vector TY in the y-axis direction is calculated according to the projection vector TY and the normal vector FYL of the y-axis, and the projection distance yty in the y-axis direction is obtained.

[0123] yty=TY·FYL;

[0124] The normal vector FZL(ZAx, ZAy, ZAz) in the z-axis direction is obtained, the projection of the projection vector TY in the z-axis direction is calculated according to the projection vector TY and the normal vector FZL of the z-axis, and the projection distance zty in the z-axis direction is obtained.

[0125] zty=TY·FZL;

[0126] Step S332: Accumulate the projection distance of the mechanical arm segment on the x-axis to obtain the total projection distance zxt on the x-axis; accumulate the projection distance of the mechanical arm segment on the y-axis to obtain the total projection distance zyt on the y-axis; accumulate the projection distance of the mechanical arm segment on the z-axis to obtain the total projection distance zzt on the z-axis, and combine the grabbing vector (Lx, Ly, Lz) to move the mechanical arm segment;

[0127] When the projection distance is less than the distance in the direction corresponding to the grabbing vector, move in the direction;

[0128] When the projection distance is greater than the distance in the direction corresponding to the grabbing vector, move in the opposite direction;

[0129] For example, if the projection distance zxt on the x-axis is less than Lx, move the mechanical arm segment in the x-axis direction; if the projection distance zxt on the x-axis is greater than Lx, move the mechanical arm segment in the direction of the plane formed by the y-axis and the z-axis; similarly, move in the y-axis and z-axis directions.

[0130] When the total projection distance zxt of the mechanical arm segment on the x-axis, the total projection distance zyt on the y-axis, and the total projection distance zzt on the z-axis are equal to the values of the grabbing vector (Lx, Ly, Lz), the grabbing posture of the mechanical arm is obtained, and the grabbing object is grabbed according to the grabbing posture. The specific grabbing steps are as follows:

[0131] Please refer to Figure 2 Step S3321: Obtain the grabbing surface of the grabbing object, extract the vertices of the grabbing surface, and record the number of vertices as g; obtain the coordinates dz (dzx, dzy, dzz) of the vertices to obtain dz1, dz2, …, dzg; calculate the center coordinates zxb (ZXx, ZXy, ZXz) of the grabbing surface according to the coordinates dz of the vertices:

[0132]

[0133] It should be noted that the grabbing surface of the grabbing object refers to the cross section of the grabbing object displayed by the robot during the grabbing process;

[0134] Step S3322: Connect the center coordinates with any two vertices to obtain vectors xla and xlb; calculate the cross product of the vectors xla and xlb to obtain the normal vector zcz (CZx, CZy, CZz) of the grabbing surface;

[0135] It should be noted that connecting any two vertices should result in a non-collinear vector;

[0136] According to the normal vector zcz(CZx, CZy, CZz) of the grabbing surface, the included angles θx, θy, and θz of the normal vector zcz(CZx, CZy, CZz) with the x-axis, the y-axis, and the z-axis are calculated, and the specific calculation process is as follows:

[0137]

[0138] Step S3323: The position of the robot end effector is obtained, the position of the robot end effector is moved to the center coordinate zxb(ZXx, ZXy, ZXz) of the grabbing surface, the grabbing angle of the robot end effector is adjusted, the included angle of the robot end effector with the x-axis is θx, the included angle of the robot end effector with the y-axis is θy, and the included angle of the robot end effector with the z-axis is θz, and grabbing is performed.

[0139] Step S3324: The neighboring object of the grabbed object is obtained, the coordinates Lzb(LZx, LZy, LZz) of the neighboring object are obtained, and the coordinates Lzb(LZx, LZy, LZz) of the neighboring object are obtained. According to the coordinates Lzb(LZx, LZy, LZz) of the neighboring object, two vertex coordinates adjacent to the grabbed object are obtained, the vertex coordinates are connected to obtain an adjacent edge, the coordinates Lzb(LZx, LZy, LZz) of the neighboring object are mapped to the adjacent edge to obtain a mapping point YSD(YSx, YSy, YSz), and the distance between the neighboring object and the adjacent edge of the grabbed object is calculated according to the mapping point YSD(YSx, YSy, YSz) and the coordinates Lzb(LZx, LZy, LZz) of the neighboring object to obtain a gap value JXZ.

[0140]

[0141] Please refer to Figure 3 ; according to the coordinates Lzb(LZx, LZy, LZz) of the neighboring object and the center coordinates zxb(ZXx, ZXy, ZXz) of the grabbing surface, a collision vector is obtained, and the included angle of the collision vector is calculated to obtain αx, αy, and αz;

[0142] The distance between the coordinates Lzb(LZx, LZy, LZz) of the neighboring object and the center coordinates zxb(ZXx, ZXy, ZXz) of the grabbing surface is calculated to obtain a grabbing depth zsd.

[0143]

[0144] The real-time position of the robot end effector is obtained, and the distance vector Mjl(MLx, MLy, MLz) between the real-time position coordinates of the robot end effector and the coordinates Lzb(LZx, LZy, LZz) of the neighboring object is obtained. According to the distance vector Mjl, the collision distance pzl is calculated.

[0145]

[0146] acquiring the position of the current robot end effector, moving towards the adjacent object according to the collision distance pzl, when the collision distance pzl=0, if the distance between the position of the robot end effector and the adjacent object is greater than the grabbing depth zsd, it is judged that grabbing can be performed;

[0147] vector sum of the collision vector and the distance vector is obtained to obtain the movement vector ydx(YDx, YDy, YDz) of the robot end effector, the obstacle is detected with the movement vector as the center line, and the obstacle coordinates are obtained; the obstacle vectors zaw1, zaw2, …, zaws are constructed according to the obstacle coordinates and the center coordinates zxb(ZXx, ZXy, ZXz) of the grabbing surface; the change angle of the robot end effector is obtained by calculating the movement vector ydx(YDx, YDy, YDz) and the obstacle vector zaw1, zaw2, …, zaws;

[0148] The included angle of the movement vector of the robot end effector is calculated to obtain the real-time angles βx, βy, βz; the grabbing angles θx, θy, θz of the robot end effector are obtained; the movement angle of the robot end effector is obtained according to the real-time angles βx, βy, βz and the grabbing angles θx, θy, θz; the movement angle of the robot end effector is compared with the change angle of the robot end effector, if the movement angle of the robot end effector is less than the change angle of the robot end effector, it is judged that movement can be performed;

[0149] The real-time angles of the robot end effector, the width mkd and the length mcd of the robot end effector are obtained; the grabbing angles θx, θy, θz of the robot end effector are combined for judgment; when the angles of the robot end effector are consistent with the grabbing angles θx, θy, θz within the change angle of the robot end effector, the width mkd of the robot end effector is less than the gap value JXZ, and the length mcd of the robot end effector is greater than the grabbing depth zsd; it is judged that grabbing can be performed.

[0150] Step S4: acquiring the characteristics of the actual object grabbed by the robot; combining the object information, the actual object grabbed by the robot is verified and judged;

[0151] Step S41: image feature extraction is performed on the actual grabbed object to obtain image information of each face of the actual grabbed object; according to the image information of each face of the actual grabbed object, the image pixels and the image contour are extracted, the image pixels of the actual grabbed object are traversed, the rgb values corresponding to the image pixels are obtained, and the rgb values are accumulated to obtain the color value of the actual grabbed object;

[0152] Obtaining the color value of the grabbing object, and calculating the color value of the grabbing object; comparing the color value of the grabbing object with the color value of the actual grabbing object;

[0153] Obtaining the image contour of the grabbing object and the image contour of the actual grabbing object, and comparing the image contour of the grabbing object with the image contour of the actual grabbing object;

[0154] According to the comparison result of the color value of the grabbing object and the color value of the actual grabbing object, and the comparison result of the image contour, obtaining the inspection result;

[0155] Step S42: According to the inspection result, judging the actual grabbing object, if judging that the actual grabbing object is consistent with the grabbing object, completing the grabbing behavior; if judging that the actual grabbing object is inconsistent with the grabbing object, excluding the actual grabbing object, and repeating the grabbing behavior according to steps S2-S3.

[0156] The above formulas are all dimensionless values calculated, and the formulas are obtained by collecting a large amount of data to simulate a formula of the nearest real situation, and the preset parameters in the formula are set by the person skilled in the art according to the actual situation, such as the weight coefficient and the proportion coefficient, the size of which is to obtain a specific value of each parameter after quantization, for subsequent comparison, and the size of the weight coefficient and the proportion coefficient only needs to not affect the proportional relationship between the parameter and the quantized value.

[0157] Finally, it should be noted that the above-described embodiments are only specific implementations of the present application, which are used to illustrate the technical solutions of the present application, but not to limit it, and the protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features, without departing from the technical range disclosed by the present application. These modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A micro-pickup control strategy for a bionic actuator based on field-of-view feedback, characterized in that, The control strategies include: Step S1: Obtain robot parameter information and visual image information; Step S2: Construct a spatial model based on visual image information; obtain the object to be grasped information, search for the object to be grasped in combination with the visual image information, obtain the position of the object to be grasped in the spatial model, and obtain the grasping position; The specific steps of step S2 are as follows: Step S21: Using visual image information, represent the visual image in three dimensions to obtain the three-dimensional data of the visual image, and construct a spatial model based on the three-dimensional data of the visual image; Step S22: Obtain the information of the object to be grasped; Based on the information of the object to be grasped, compare the visual image to obtain the image position of the object to be grasped, and obtain the spatial position of the object to be grasped in the spatial model based on the image position of the object to obtain the grasping position (Zx, Zy, Zz). The specific steps of step S21 are as follows: Step S211: Obtain the current position as the origin (0, 0, 0), define the horizontal rightward direction as the x-axis, the horizontal downward direction as the y-axis, and the direction outward along the optical axis as the z-axis; construct a three-dimensional coordinate system based on the origin position, x-axis, y-axis, and z-axis. Step S212: Based on the visual image information and the three-dimensional coordinate system, calculate the three-dimensional coordinates of the image pixels and construct a spatial model from the three-dimensional coordinates of the image pixels. The specific steps of step S212 are as follows: From the visual image information, the depth information of the visual image is obtained. Based on the depth information of the visual image, the depth corresponding to the image pixel is obtained, and the coordinate Sz corresponding to the image pixel in the z-axis direction is obtained. Based on the intersection of the pixel coordinate system and the optical axis, obtain the optical center coordinates (Cx, Cy); obtain the focal length (Fx, Fy). Obtain the position u of the image pixel in the horizontal direction of the pixel coordinate system and the position v of the image pixel in the vertical direction of the pixel coordinate system. Combine the optical center coordinates (Cx, Cy), focal length (Fx, Fy) and the coordinate Sz of the image pixel on the z-axis, and calculate the coordinate Sx of the image pixel in the x-axis direction and the coordinate Sy of the image pixel in the y-axis direction. Based on the coordinates Sz (z-axis), Sx (x-axis), and Sy (y-axis) of the image pixel, the three-dimensional coordinates (Sx, Sy, Sz) of the image pixel are obtained; based on the three-dimensional coordinates of the image pixel, the space is filled to obtain the spatial model. Acquire point cloud data of objects in the space. Based on the point cloud data, obtain the object point coordinates (dbx, dby, dbz) and the corresponding reflected light intensity. Based on the object point coordinates (dbx, dby, dbz) in the point cloud data, obtain the reflected light intensity of the corresponding point coordinates in the spatial model. Compare the reflected light intensity of the point coordinates in the point cloud data with the reflected light intensity of the corresponding point coordinates in the spatial model. If the two reflected light intensities are different, replace the object in the spatial model according to the object point coordinates in the point cloud data and optimize the spatial model. Step S3: Move the robot using the spatial model and the grasping position to obtain the robot's current position and initial position. Based on the initial position and the grasping position, obtain the robot's grasping path. Detect nearby objects of the grasped object using the spatial model to obtain the gap between the grasped object and nearby objects. Combine the robot's parameter information and the grasping path to calculate the grasping angle. Detect the robot's grasping path based on the grasping angle and the gap between the grasped object and nearby objects. Judge the grasping behavior based on the detection results and control the robot's grasping behavior. Step S4: Obtain the characteristics of the object actually grasped by the robot; combine the object information to verify and judge the object actually grasped by the robot.

2. The micro-pickup control strategy for a bionic actuator based on field-of-view feedback as described in claim 1, characterized in that, The specific steps of step S1 are as follows: Step S11: Obtain the number of joints n of the robot arm; Measure the length cd of the robot arm segment according to the number of joints, count the lengths of the robot arm segments, and obtain the list of robot arm segments lbc, lbc = [cd(1), cd(2) ..., cd(n-1)]; Step S12: Acquire pixel information of the visual image of the space where the robot is located and construct a pixel coordinate system; represent the pixel position in the visual image using (u, v); acquire the depth information of the visual image; the pixel information and the depth information of the visual image constitute the visual image information.

3. The micro-pickup control strategy for a bionic actuator based on field-of-view feedback according to claim 1, characterized in that, The specific steps of step S3 are as follows: Step S31: Move the robot using the spatial model and the gripping position. Move the robot towards the gripping position. When the robot cannot move, obtain the three-dimensional coordinates of the robot's current position to get the initial position (Jx, Jy, Jz). Connect the initial position and the gripping position to get the gripping vector (Lx, Ly, Lz). Use the gripping vector as the robot's gripping path. Step S32: Based on the robot parameter information, obtain the list of robotic arm segments lbc, lbc = [cd(1), cd(2) ..., cd(n-1)]; Combine the robot's grasping path to determine the grasping behavior; Step S33: Fit the robotic arm segment to the robot's grasping path to obtain the grasping posture of the robotic arm segment, and grasp the object according to the grasping posture.

4. The bionic actuator micro-pickup control strategy based on field-of-view feedback according to claim 3, characterized in that, The specific steps of step S32 are as follows: Step S321: Based on the list of robotic arm segments lbc = [cd(1), cd(2), ..., cd(n-1)], calculate the longest grasping distance of the robotic arm to obtain the longest distance ZCJ: ； Step S322: Based on the robot's grasping path, obtain the grasping vector (Lx, Ly, Lz). Calculate the distance between the robot and the grasped object based on the grasping vector to obtain the grasping distance ZQJ. ； Step S323: Determine whether the robot can grasp the object based on the longest distance ZCJ and the grasping distance ZQJ. If ZCJ≥ZQJ, it indicates that the robot is able to grasp the object; proceed to step S33; If ZCJ < ZQJ, it indicates that the robot is unable to grasp the object and will issue an alarm error.

5. The bionic actuator micro-pickup control strategy based on field-of-view feedback according to claim 3, characterized in that, The specific steps of step S33 are as follows: Step S331: Obtain the real-time position of the robotic arm segment, and calculate the projection of the robotic arm segment onto the three-dimensional coordinate system based on the real-time position of the robotic arm segment. The specific calculation is as follows: Obtain the joint coordinates ZB1 (Z1x, Z1y, Z1z) and ZB2 (Z2x, Z2y, Z2z) of adjacent segments of the robotic arm, and calculate the projection vector TY (Z1x-Z2x, Z1y-Z2y, Z1z-Z2z) based on the joint coordinates. Obtain the normal vector FXL (XAx, XAy, XAz) along the x-axis. Calculate the projection of the projection vector TY onto the x-axis based on the projection vector TY and the normal vector FXL along the x-axis, and obtain the projection distance xty along the x-axis. Similarly, the projected distance yty on the y-axis and the projected distance zty on the z-axis are obtained; Step S332: Accumulate the projected distances of the robotic arm segments on the x-axis to obtain the total projected distance zxt on the x-axis; accumulate the projected distances of the robotic arm segments on the y-axis to obtain the total projected distance zyt on the y-axis; accumulate the projected distances of the robotic arm segments on the z-axis to obtain the total projected distance zzt on the z-axis; combine the grasping vector (Lx, Ly, Lz) to move the robotic arm segments; When the projected distance is less than the distance in the direction corresponding to the grab vector, move in that direction; When the projected distance is greater than the distance in the direction corresponding to the grab vector, move in the opposite direction; When the total projection distance zxt of the robotic arm segment on the x-axis, the total projection distance zyt on the y-axis, and the total projection distance zzt on the z-axis are equal to the values ​​of the grasping vector (Lx, Ly, Lz), the grasping posture of the robotic arm is obtained, and the grasping object is grasped according to the grasping posture.

6. The micro-pickup control strategy for a bionic actuator based on field-of-view feedback according to claim 5, characterized in that, The specific steps of step S332 are as follows: Obtain the grabbing surface of the object to be grabbed, extract the vertices of the grabbing surface, and denote the number of vertices as g; obtain the coordinates dz(dzx, dzy, dzz) of the vertices to get dz1, dz2, ..., dzg; calculate the center coordinates zxb(ZXx, ZXy, ZXz) of the grabbing surface based on the coordinates dz of the vertices: ； Connect the center coordinates to any two vertices to obtain vectors xla and xlb; perform a cross product calculation on vectors xla and xlb to obtain the vertical vector zcz (CZx, CZy, CZz) of the grabbing surface. Calculate the angles θx, θy, and θz between the vertical vector zcz (CZx, CZy, CZz) of the grasping surface and the x-axis, y-axis, and z-axis, respectively. Obtain the position of the robot's end effector, move the robot's end effector to the center coordinates of the gripping surface, adjust the angle of the robot's end effector so that its angle with the x-axis is θx, its angle with the y-axis is θy, and its angle with the z-axis is θz, and then perform the gripping. The system acquires the coordinates of neighboring objects of the target object, Lzb(LZx, LZy, LZz). Based on the coordinates of the neighboring objects, it acquires the coordinates of the two nearest vertices of the target object and connects the vertex coordinates to obtain the adjacent edge. The coordinates of the neighboring objects, Lzb(LZx, LZy, LZz), are mapped to the adjacent edge to obtain the mapping point YSD(YSx, YSy, YSz). Based on the coordinates of the mapping point and the neighboring objects, the distance between the neighboring objects and the adjacent edge of the target object is calculated to obtain the gap value JXZ. ； Based on the coordinates of nearby objects and the center coordinates of the grabbing surface, the collision vector is obtained, and the angle between the collision vectors is calculated to obtain αx, αy, and αz.

7. The micro-pickup control strategy for a bionic actuator based on field-of-view feedback according to claim 6, characterized in that, Step S332 further includes: The distance between the coordinates of nearby objects and the center coordinates of the gripping surface is calculated to obtain the gripping depth zsd; ； Obtain the distance vector Mjl(MLx,MLy,MLz) between the robot's end effector and the coordinates Lzb(LZx,LZy,LZz) of a neighboring object; calculate the collision distance pzl based on the distance vector Mjl. ； Obtain the current position of the robot's end effector, and move it towards a nearby object based on the collision distance pzl. When the collision distance pzl = 0, if the distance between the robot's end effector and the nearby object is greater than the grasping depth zsd, it is determined that grasping can be performed.

8. The micro-pickup control strategy for a bionic actuator based on field-of-view feedback according to claim 7, characterized in that, Step S332 further includes: The collision vector and distance vector are summed to obtain the movement vector ydx(YDx, YDy, YDz) of the robot's end effector. Using the movement vector as the center line, obstacles are detected and their coordinates are obtained. Based on the obstacle coordinates and the center coordinates zxb(ZXx, ZXy, ZXz) of the gripping surface, obstacle vectors zaw1, zaw2, ..., zaws are constructed. The changing angle of the robot's end effector is calculated based on the movement vector ydx(YDx, YDy, YDz) and the obstacle vectors zaw1 to zaws. The angles between the movement vectors of the robot end effector are calculated to obtain real-time angles βx, βy, and βz; the grasping angles θx, θy, and θz of the robot end effector are obtained; based on the real-time angles βx, βy, and βz and the grasping angles θx, θy, and θz, the movement angle of the robot end effector is obtained; the movement angle of the robot end effector is compared with the change angle of the robot end effector; if the movement angle of the robot end effector is less than the change angle of the robot end effector, it is determined that movement can be performed. Obtain the real-time angle, width (mkd), and length (mcd) of the robot end effector; combine this with the grasping angles (θx, θy, θz) of the robot end effector to make a judgment; if the robot end effector's angle matches the grasping angles (θx, θy, θz) within the changing angle range, and the robot end effector's width (mkd) is less than the gap value (JXZ), and the robot end effector's length (mcd) is greater than the grasping depth (zsd), then it is determined that grasping can be performed.

Images