Mechanical arm control method

Abstract

The invention discloses a mechanical arm control method, and relates to the technical field of material conveying, and the method comprises the following steps: carrying out part target detection on a material conveying image to obtain a target part image; performing preset linear pose fitting on the target part image to obtain part pose information; inverse motion solving is carried out according to the part position information and the part pose information corresponding to the material transportation image, and mechanical arm motion information is obtained; and controlling the mechanical arm to convey the parts based on the mechanical arm motion information. According to the method, pose detection is carried out on the target part image obtained after target detection is carried out on the collected material transportation image, part pose information is obtained, inverse kinematics solution is carried out on the basis of the part position information and the part pose information contained in the material transportation image, and mechanical arm motion information corresponding to mechanical arm grabbing and mechanical arm delivery is determined; precise part grabbing and part transferring are carried out based on mechanical arm movement information, and the part conveying timeliness is guaranteed.

Description

Technical Field

[0001] The present application relates to the technical field of material conveying, and in particular to a robot arm control method. Background Art

[0002] A challenge faced in the current assembly line production process is how to handle materials that are scattered and of different shapes and sizes. At present, the transportation of these materials mainly relies on manual labor, which not only leads to high costs and low efficiency, but also its reliability is difficult to be effectively guaranteed. Although there have been some attempts to reduce manual participation in the material transportation process by introducing robotic arms, unfortunately, these robotic arms can usually only perform grasping operations in specific scenarios, and they still have limitations in accurately delivering materials. In addition, for different types of parts, the gripping accuracy of the robotic arms is often not high enough, which further reduces the efficiency of material supply.

[0003] In view of this, how to effectively control the robot arm to ensure the timeliness and efficiency of material supply has become an urgent problem to be solved. Therefore, it is necessary to develop a more intelligent and flexible robot arm control system that can not only adapt to a variety of materials and environmental conditions, but also improve the gripping accuracy, thereby significantly improving the efficiency and reliability of the entire material conveying process. Summary of the invention

[0004] The main purpose of this application is to provide a robot arm control method, aiming to solve the technical problem of how to effectively control the robot arm to ensure the timeliness and efficiency of material supply.

[0005] To achieve the above objectives, the present application proposes a robot arm control method, the method comprising:

[0006] Perform part target detection on the material transportation image to obtain the target part image;

[0007] Performing a preset linear pose fitting on the target part image to obtain part pose information;

[0008] Perform inverse kinematics solution based on the part position information corresponding to the material transportation image and the part posture information to obtain the robot arm motion information;

[0009] The robot arm is controlled to transport parts based on the robot arm motion information.

[0010] In one embodiment, the step of performing a preset linear pose fitting on the target part image to obtain part pose information includes:

[0011] Performing a preset linear fitting on the target part image to obtain target principal axis information;

[0012] Determine the target segmentation center of gravity distance according to the center point information corresponding to the target main axis information;

[0013] The part pose information is determined based on the target segmentation center of gravity distance.

[0014] In one embodiment, the step of performing a preset linear fitting on the target part image to obtain target spindle information includes:

[0015] Segmenting the target part image and detecting the center of gravity to obtain the target center of gravity coordinates;

[0016] The target centroid coordinates are linearly fitted by the least square method to obtain the target principal axis information.

[0017] In one embodiment, the step of determining the target segmentation center of gravity distance according to the center point information corresponding to the target main axis information includes:

[0018] Performing edge fitting on the target principal axis information to obtain the target edge center of gravity;

[0019] The target segmentation center of gravity distance is determined according to the center point information corresponding to the target edge center of gravity point and the target main axis information.

[0020] In one embodiment, the step of determining part pose information based on the target segmentation center of gravity distance includes:

[0021] Determine the short axis direction corresponding to the target part according to the target segmentation center of gravity distance;

[0022] Determine the target rotation angle based on the angle between the straight line equation corresponding to the short axis direction and the horizontal axis;

[0023] The part position and posture information is determined according to the target rotation angle.

[0024] In one embodiment, the step of performing part target detection on the material transportation image to obtain the target part image includes:

[0025] Use the pre-trained YOLOv5 network to detect parts in material transportation images and obtain initial part images.

[0026] Edge contour extraction is performed on the initial part image to obtain a target part image.

[0027] In one embodiment, the machine extracts edge contours from the initial part image to obtain a target part image, comprising:

[0028] Performing Laplacian Gaussian edge detection on the initial part image to obtain target edge information;

[0029] Performing pixel grayscale traversal on the initial part image according to the target edge information, and taking pixel points that meet the preset edge grayscale standard as target edge points;

[0030] The initial part image is segmented based on the target edge points to obtain a target part image.

[0031] In addition, to achieve the above-mentioned purpose, the present application also proposes a robot arm control device, the robot arm control device comprising:

[0032] An image detection module is used to detect target parts in material transportation images and obtain target part images;

[0033] A posture fitting module, used to perform a preset linear posture fitting on the target part image to obtain part posture information;

[0034] An inverse kinematics analysis module, used to perform inverse kinematics analysis based on the part position information corresponding to the material transport image and the part posture information to obtain the robot arm motion information;

[0035] The transport control module is used to control the robot arm to transport parts based on the movement information of the robot arm.

[0036] In addition, to achieve the above-mentioned purpose, the present application also proposes a robotic arm control device, which includes: a memory, a processor, and a robotic arm control program stored in the memory and executable on the processor, and the robotic arm control program is configured to implement the steps of the robotic arm control method as described above.

[0037] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, which stores a program for implementing the robot arm control method, and the program for implementing the robot arm control method is executed by a processor to implement the steps of the robot arm control method as described above.

[0038] The present application provides a method for controlling a robotic arm. The method includes: performing part target detection on a material transport image to obtain a target part image; performing preset linear pose fitting on the target part image to obtain part pose information; performing inverse motion solution based on part position information and part pose information corresponding to the material transport image to obtain robotic arm motion information; and controlling the robotic arm to transport parts based on the robotic arm motion information.

[0039] The present application combines material transportation images collected during the parts delivery process to detect whether the target parts are included therein. If so, posture detection is performed based on the target part image obtained after target detection to obtain part posture information. Then, based on the part position information and part posture information contained in the target part space in the material transportation image, inverse kinematics is performed to determine the robotic arm motion information corresponding to the robotic arm grasping and robotic arm movement, thereby accurately grasping and transferring parts based on the robotic arm motion information to ensure the timeliness of part delivery in the production process. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings herein are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the description, are used to explain the principles of the present application.

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0042] Figure 1 This is a flow chart of the first embodiment of the robot arm control method of the present application;

[0043] Figure 2 This is a schematic diagram of edge pixel traversal of the first embodiment of the robot arm control method of the present application;

[0044] Figure 3 This is a schematic diagram of a robotic arm control process of the first embodiment of the robotic arm control method of the present application;

[0045] Figure 4 This is a flow chart of the second embodiment of the robot arm control method of the present application;

[0046] Figure 5 This is a schematic diagram of a target part gravity center determination process of the second embodiment of the robot arm control method of the present application;

[0047] Figure 6 A brief flow chart of the robot arm control method of the present application is provided;

[0048] Figure 7 A process schematic diagram of the robot arm control method of the present application is provided;

[0049] Figure 8 This is a schematic diagram of the module structure of the robot arm control device according to an embodiment of the present application;

[0050] Fig. 9 Schematic diagram of the device structure of the hardware operating environment involved in the robot arm control method in the embodiment of the present application.

[0051] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0053] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0054] The main solution of this application is: perform part target detection on the material transportation image to obtain the target part image; perform preset linear pose fitting on the target part image to obtain part pose information; perform inverse motion solution based on the part position information and part pose information corresponding to the material transportation image to obtain the robot arm motion information; and control the robot arm to transport parts based on the robot arm motion information.

[0055] The existing technology for handling materials of different shapes and sizes that are scattered mainly relies on manual labor, which not only leads to high costs and low efficiency, but also its reliability is difficult to be effectively guaranteed. Although attempts are currently being made to introduce robotic arms to reduce manual involvement in the material transportation process, unfortunately, these robotic arms can usually only perform grasping operations in specific scenarios, and the gripping accuracy of the robotic arms is often not high enough for different types of parts, which further reduces the efficiency of material supply. How to solve the problem of inaccurate and untimely material supply during the production process and ensure the timeliness of transportation during the production process has become an urgent problem to be solved.

[0056] In the face of the problem of accurate gripping of different parts by robots, this application studies computer vision technology, combines sensor technology and adaptive control technology, studies the adaptive grasping algorithm of high-precision robotic arms, and completes the precise acquisition and delivery of materials of different sizes by robotic arms. Specifically, this application detects whether the target parts are contained in the material transportation images collected during the parts transportation process. When they exist, posture detection is performed based on the target part image obtained after the target detection to obtain the part posture information. Then, based on the part position information and part posture information in the target part space contained in the material transportation image, inverse kinematics is performed to determine the robotic arm motion information corresponding to the robotic arm grasping and the robotic arm movement, thereby accurately grasping and delivering parts based on the robotic arm motion information, solving the problem of inaccurate and untimely material supply in the production process and ensuring the timeliness of part delivery in the production process.

[0057] It should be noted that the execution subject of this embodiment can be a robot control system, or a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a robot control device capable of realizing the above functions, etc., and this embodiment does not specifically limit this. The following takes the robot control device (referred to as the control device) as the execution subject as an example to illustrate this embodiment and the following embodiments.

[0058] Based on this, the present application embodiment provides a method for controlling a robotic arm, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the robot arm control method of the present application.

[0059] In this embodiment, the robot arm control method includes steps S10 to S40:

[0060] Step S10, performing part target detection on the material transportation image to obtain a target part image;

[0061] It is easy to understand that the above-mentioned material transportation image is the image collected during the transportation of parts on the assembly line. In this embodiment, an RGBD camera can be used as a sensor to perceive the surrounding environment information to collect material transportation images. Therefore, the material transportation image is an RGBD image, which combines an RGB image and a depth image, wherein the RGB image contains the color information of the object, and the depth image provides the spatial information of the object. In this embodiment, the material transportation image can be segmented and detected for the target part to determine whether the target part exists in the assembly line. If so, the image is segmented to obtain the target part image corresponding to the target part.

[0062] In a feasible implementation manner, in this embodiment, step S10 may include steps A1 to A2:

[0063] Step A1, performing part target detection on the material transportation image through the pre-trained YOLOv5 network to obtain an initial part image;

[0064] Step A2: extract edge contours from the initial part image to obtain a target part image.

[0065] It is easy to understand that this embodiment uses the pre-trained YOLOv5 network to perform part target detection on the objects contained in the material transportation image, and extracts the target detection sample frame output by the YOLOv5 network in batches to obtain a cropped image of the detected target part, that is, the initial part image. Then the control device can refine the edge contour of the initial part to obtain an accurate target part image.

[0066] In a feasible implementation manner, in this embodiment, step A2 may include steps A21 to A23:

[0067] Step A21, performing Gaussian Laplace edge detection on the initial part image to obtain target edge information;

[0068] Step A22, performing pixel grayscale traversal on the initial part image according to the target edge information, and taking pixel points that meet the preset edge grayscale standard as target edge points;

[0069] Step A23, performing image segmentation on the initial part image based on the target edge points to obtain a target part image.

[0070] It is easy to understand that the control device can scan the initial part image after preliminary cropping, divide it into n images of the same size by column or row, and then use the segmentation image edge point detection algorithm to extract the object contour, and regard the points with large grayscale changes determined in the detection results as the contour points of the object, that is, the target edge points, and then refine the part edge based on the target edge points to obtain the target part image.

[0071] Specifically, in order to accurately extract feature point information at the edge of the image (target contour), the segmented image edge point detection algorithm in this embodiment can first obtain edge coordinate information of the initial part image based on Gaussian Laplacian edge detection, that is, target edge information.

[0072] Then, according to the target edge information, the initial part image after preliminary processing is traversed, and the 16 pixel points with the same pixel value as the edge point center p on the circle with a radius of 3 pixels on the edge of the image are counted. The number of pixel points that form a connected area with the edge point center p through the 8-adjacent method can be shown as follows: Figure 2 As shown, Figure 2 This is a schematic diagram of edge pixel traversal of the first embodiment of the robot arm control method of the present application.

[0073] At the same time, this embodiment pre-sets a grayscale threshold T during the edge feature point detection process. If the number of pixel points with a larger grayscale difference with the edge center point among the adjacent pixel points of an edge point is not less than T, the edge point is considered to meet the preset edge grayscale standard and is a target edge point; if it is less than T, it means that the grayscale value difference between the edge point and the adjacent similar points is not obvious, and it will not be judged as a target edge point.

[0074] Step S20, performing preset linear pose fitting on the target part image to obtain part pose information;

[0075] Step S30, performing inverse kinematics solution according to the part position information corresponding to the material transportation image and the part posture information to obtain the robot arm motion information;

[0076] Step S40: Controlling the robot arm to transport parts based on the robot arm motion information.

[0077] It can be understood that after obtaining the precisely segmented target part image, the present embodiment can perform a linear fitting analysis on its inner center of gravity to obtain the posture information between each target part and the horizontal axis, that is, the part posture information, and then perform inverse motion solution based on the part posture information and the spatial three-dimensional position information contained in the material transportation image, that is, the above-mentioned part position information, to obtain the angle control information of each joint of the robot arm during the part clamping and part conveying process, that is, the above-mentioned robot arm motion information.

[0078] Specifically, this embodiment can perform inverse kinematics analysis based on the UR5 robot arm to calculate the joint angles when the end effector moves to the target point corresponding to the part position information and the part posture information, and establish the following equation:

[0079] -sin(θ)p x +cos(γ)p y =d; (1)

[0080] Among them, let p x =ρcos(φ), p y =ρsin(φ), and φ=Atan2(p y ,p x ), and substitute it into formula (1), we get:

[0081]

[0082] Then we can get:

[0083]

[0084] The grasping control joint angle θ corresponding to the robotic arm can be expressed as:

[0085]

[0086] It is easy to understand that, in addition to the above-mentioned grasping control joint angle, the robot arm motion information in this embodiment may also include the robot arm's optimal motion trajectory information from the part grasping point to the target delivery point, thereby further improving the part transportation efficiency.

[0087] It should be understood that in the present embodiment, an intercepting grasping strategy can be adopted in the process of parts transportation of the robot arm, that is, the control device can first send the grasping control joint angle to the corresponding robot arm, and at the same time estimate the waiting time of the robot arm for the target part to move to the lower end of the robot arm and just merge with the robot arm according to the collection point of the material transportation image, the motion state of the assembly line and the movement speed of the robot arm. Then, when the control device predicts that the target part will arrive at the lower part of the tool end of the robot arm according to the waiting time of the robot arm, the control device controls the robot arm to grasp the target part based on the grasping control joint angle, thereby reducing the movement path of the robot arm and improving the grasping efficiency.

[0088] For ease of understanding, Figure 3 Take the example of the robot arm control process of this embodiment as an example. Figure 3 This is a schematic diagram of the robotic arm control process of the first embodiment of the robotic arm control method of the present application.

[0089] like Figure 3 As shown, in this embodiment, the target part can be detected by the RGBD camera and the image information can be returned to the main control board (such as Figure 3 The main control board processes the collected material transportation images, detects the target parts and estimates the pose of the parts; at the same time, it performs inverse kinematics based on the part position information and part pose information corresponding to the material transportation images, calculates the angles of each joint of the end effector of the robot arm at the target node, and sends the angle information to the control board that controls the movement of the robot arm (such as Figure 3 After receiving the angle information, the control board controls the servo drive and the robotic arm to complete the part grabbing action. At the same time, during the whole process, the Hu Kong board will send the camera image information, log information, and ROS topic publishing information to the PC for real-time display, so that the operator can know the progress of the robotic arm transportation.

[0090] In this embodiment, when the robot arm faces different parts that need to be processed, the RGBD camera can be used to quickly detect the parts in the environment, and the center point position and posture angle of each part can be calculated based on the detection results, the posture of the part can be estimated, and the posture information can be solved into the robot arm drive information for grasping. At the same time, this embodiment uses the YOLOv5 network and the segmented image edge point detection algorithm that take into account both lightness and accuracy, and directly obtains depth information through the RGBD camera to calculate the position of the part in three-dimensional space, which ensures the real-time performance of the system while also improving the accuracy of grasping.

[0091] Therefore, in the face of the problem of accurate gripping of different parts by robots, this embodiment studies the adaptive grasping algorithm of high-precision robotic arms based on computer vision technology, combined with sensor technology and adaptive control technology, and completes the accurate acquisition and delivery of materials of different sizes by robotic arms. Specifically, this application detects whether the target parts are contained in the material transportation images collected during the parts transportation process. When they exist, posture detection is performed based on the target part image obtained after the target detection to obtain the part posture information. Then, based on the part position information and part posture information of the target part space contained in the material transportation image, inverse kinematics is solved to determine the robotic arm motion information corresponding to the robotic arm grasping and the robotic arm movement, so as to accurately grasp and transfer parts based on the robotic arm motion information, solve the problem of inaccurate and untimely material supply in the production process, and ensure the timeliness of part delivery in the production process.

[0092] This embodiment provides a robot arm control method, which includes: performing part target detection on a material transport image through a pre-trained YOLOv5 network to obtain an initial part image; performing Gaussian Laplace edge detection on the initial part image to obtain target edge information; performing pixel grayscale traversal on the initial part image according to the target edge information, and taking pixel points that meet the preset edge grayscale standard as target edge points; performing image segmentation on the initial part image based on the target edge points to obtain a target part image. Performing preset linear pose fitting on the target part image to obtain part pose information; performing inverse motion solution based on the part position information and part pose information corresponding to the material transport image to obtain robot arm motion information; and controlling the robot arm to transport parts based on the robot arm motion information. This embodiment detects whether the material transportation image collected during the parts transportation process contains the target parts. If so, posture detection is performed based on the target part image obtained after target detection to obtain part posture information. Then, inverse kinematics is performed based on the part position information and part posture information of the target part space contained in the material transportation image to determine the robot arm motion information corresponding to the robot arm grasping and robot arm movement, thereby accurately grasping and transferring parts based on the robot arm motion information, solving the problem of inaccurate and untimely material supply in the production process and ensuring the timeliness of part transportation in the production process.

[0093] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction and will not be repeated later.

[0094] Based on the first embodiment, please refer to Figure 4 , Figure 4 This is a flow chart of the second embodiment of the robot arm control method of the present application. In this embodiment, step S20 includes steps B1 to B3:

[0095] Step B1, perform a preset linear fitting on the target part image to obtain target main axis information;

[0096] It should be understood that in this embodiment, linear fitting can be performed according to the centroid information of the target part image to obtain target main axis information representing the horizontal axis and vertical axis of the target part.

[0097] In a feasible implementation manner, in this embodiment, step B1 includes steps B11 to B12:

[0098] Step B11, perform segmented centroid detection on the target part image to obtain target centroid coordinates;

[0099] Step B12, perform linear fitting on the target centroid coordinates by the least squares method to obtain target main axis information.

[0100] It can be understood that in this embodiment, the target part image can be divided into multiple sub-part images by row segmentation or column segmentation. Then, first, based on the starting coordinates and ending coordinates of the targets in each column of the sub-part images after column segmentation, calculate the centroid coordinates of the targets within the column, that is, the target centroid coordinates; then, take the centroid coordinates of each column as a set of data, and use the least squares method to perform linear fitting to obtain the first linear equation representing the horizontal main axis position of the sub-part:

[0101] f1(x) = a1x + b1; (5)

[0102] Where x is the abscissa of the centroid of each column, and a1, b1 are the linear fitting parameters.

[0103] Similarly, in this embodiment, the centroid coordinates of the targets within each row can also be calculated according to the starting coordinates and ending coordinates of the targets in each row of the sub-part images after row segmentation of the target part image. Then, take the centroid coordinates of each row as a set of data, and use the least squares method to perform linear fitting to obtain the second linear equation representing the vertical main axis position of the sub-part:

[0104] f2(y) = a2y + b2; (6)

[0105] Where y is the ordinate of the centroid of each row, and a2, b2 are the linear fitting parameters.

[0106] It should be noted that in the above least squares method operation process, for n centroid points (xi, yi) (0 < i ≤ n) in each row or each column, assuming their fitting straight line is Y = aχ + b, then for each xi, its fitting value is Y i = ax i + b, so the objective function can be:

[0107]

[0108] At this time, a and b corresponding to when the above f(x) reaches the minimum value are the specific parameter values ​​of the linear fitting straight line equation.

[0109] It can be understood that the first straight line equation and the second straight line equation are the target spindle information obtained when the target part image is segmented by columns and by rows.

[0110] Step B2, determining the target segmentation center of gravity distance according to the center point information corresponding to the target main axis information;

[0111] It is easy to understand that after determining the target spindle information, this embodiment needs to determine the edge contour of the target part based on the target spindle information, and then determine the center of gravity of the target part.

[0112] In a feasible implementation manner, in this embodiment, step B2 includes steps B21 to B22:

[0113] Step B21, performing edge fitting on the target principal axis information to obtain the target edge center of gravity;

[0114] Step B22, determining the target segmentation center of gravity distance according to the center point information corresponding to the target edge center of gravity point and the target main axis information.

[0115] It is easy to understand that in this embodiment, the center point coordinates of the target part can be determined according to the intersection of each equation in the target spindle information, and the edge fitting of the target spindle information can be performed to obtain the target edge center of gravity point. The specific effect can be as follows: Figure 5 As shown, Figure 5 This is a schematic diagram of the target part gravity center determination process of the second embodiment of the robot arm control method of this application. Figure 5 As shown, after edge fitting of f1 and f2, four edge centroid points point_left, point_right, point_top and point_bottom can be obtained.

[0116] Then, this embodiment can obtain the horizontal distances d2left and d2right from the first and last centroid points point_left (xleft, yleft) and point_right (xright, yright) obtained by column segmentation to the center point coordinates.

[0117] Similarly, calculate the vertical distances d1top and d1bottom from the first two centroid points point_top (xtop, ytop) and point_bottom (xbottom, ybottom) obtained by row segmentation to the center point coordinates. At this time, d2left, d2right, d1top and d1bottom are the centroid distances of the target segmentation mentioned above.

[0118] Step B3, determining part pose information based on the target segmentation center of gravity distance.

[0119] It is understandable that after determining the target segmentation distance, this embodiment can determine the part posture information of the target part with a suitable grasping angle and grasping force according to the center of gravity information of the target part.

[0120] In a feasible implementation manner, in this embodiment, step B3 includes steps B31 to B33:

[0121] Step B31, determining the short axis direction corresponding to the target part according to the target segmentation center of gravity distance;

[0122] Step B32, determining a target rotation angle based on an angle between a straight line equation corresponding to the short axis direction and a horizontal axis;

[0123] Step B33, determining the part posture information according to the target rotation angle.

[0124] Specifically, this embodiment can first determine the short axis direction of the target part based on the distance between the vertical and horizontal lines and the center point in the target segmentation center of gravity distance, that is, the main axis direction corresponding to the shortest distance in the above target segmentation center of gravity distance is used as the short axis direction. It is easy to understand that Figure 5 The distance from point_top (xtop, ytop) to the center point is the shortest, so f2 is in the direction of the minor axis.

[0125] At this time, the angle between the direction of the straight line equation corresponding to the short axis and the horizontal axis can be the target rotation angle of the target part. The calculation expression of the target rotation angle is as follows:

[0126]

[0127] Among them, (xm1, ym1 and (xm2, ym2) are the intersection points with the target part contour in the minor axis direction.

[0128] It should be understood that after obtaining the target rotation angle, the control device can combine the image and depth information returned by the RGBD camera to obtain the spatial three-dimensional position information of the target, calculate the center point coordinates and posture angles to calculate the actual posture of the target part in the three-dimensional space, and thus determine the robot arm joint angle information used to estimate the appropriate grasping angle and grasping force of the target part.

[0129] To sum up, when the robot arm needs to process different parts, this embodiment can quickly detect the parts in the environment based on the images collected by the RGBD camera to obtain the target part image, and calculate the center point position and posture angle of each part based on the detection results, estimate the posture of the part, and solve the posture information into robot arm driving information for grasping.

[0130] Therefore, this embodiment can design an adaptive grasping algorithm for the robot arm based on the appearance features of the parts extracted from the image, so as to enable the robot arm to accurately obtain materials of different sizes, allowing the robot arm to accurately clamp parts of different sizes and improve the efficiency of material supply in the production process.

[0131] The present embodiment discloses segmenting the target part image for center of gravity detection to obtain the target center of gravity coordinates; performing linear fitting on the target center of gravity coordinates by the least squares method to obtain the target main axis information. Perform edge fitting on the target main axis information to obtain the target edge center of gravity point; determine the target segmentation center of gravity distance based on the target edge center of gravity point and the center point information corresponding to the target main axis information. Determine the short axis direction corresponding to the target part based on the target segmentation center of gravity distance; determine the target rotation angle based on the angle between the straight line equation corresponding to the short axis direction and the horizontal axis; determine the part posture information based on the target rotation angle. When the robot arm faces different parts that need to be processed, this embodiment can quickly detect the parts in the environment based on the image collected by the RGBD camera to obtain the target part image, and calculate the center point position and posture angle of each part based on the detection results, estimate the posture of the part, and solve the posture information into the robot arm drive information for grasping.

[0132] Therefore, this embodiment can design an adaptive grasping algorithm for the robot arm based on the appearance features of the parts extracted from the image, so as to enable the robot arm to accurately obtain materials of different sizes, allowing the robot arm to accurately clamp parts of different sizes and improve the efficiency of material supply in the production process.

[0133] For example, in order to help understand the technical concept or technical principle of the robot arm control method after the present embodiment is combined with the above-mentioned embodiment 1 and embodiment 2, please refer to Figure 6 and Figure 7 , Figure 6 A brief flow chart of the robot arm control method of the present application is provided. Figure 7 A process schematic diagram of the robot arm control method of the present application is provided, as follows:

[0134] like Figure 6 As shown, the first step of the present application for controlling the robotic arm is to obtain an image from the camera, wherein the RGB image is used to extract image features after processing, and the depth image is used to obtain the depth information of the image;

[0135] The second step is to detect the image, identify the parts in the original camera image, and determine whether there are target parts in the image;

[0136] The third step is to crop the target part from the original camera image. Specifically, the part contour is extracted by segmenting the image edge point detection algorithm.

[0137] The fourth step is to use the extracted contour to calculate the pose information of the part in three-dimensional space and obtain the spatial three-dimensional information of the target part. Specifically, the linear equation f1 of the horizontal axis of the part is calculated based on the part contour, and the linear equation f2 of the vertical axis of the part is calculated to calculate the pose angle of the part;

[0138] The fifth step is to perform inverse kinematics on the part posture information and the three-dimensional spatial information to obtain the robot arm drive information. Specifically, the coordinate and angle information are converted into position information and joint angles.

[0139] The sixth step is to perform robotic arm grasping based on the robotic arm drive information.

[0140] Specifically, based on Figure 7 It can be seen that the specific processing flow of this application is divided into four steps: target detection, pose estimation, motion planning and robotic arm grasping. Therefore, this application first performs image detection based on the RGBD image, then extracts the object contour, and then calculates the center of gravity of the segmented image after the extracted contour is refined, fits the straight line equation according to the center of gravity, obtains the horizontal main axis straight line equation f1 and the vertical main axis straight line equation f2, then calculates the object center point, and then calculates the pose angle, and finally obtains the object pose;

[0141] Then, the spatial three-dimensional information is obtained based on the base coordinates of the RGBD image, that is, Figure 7 The position information in the image and the joint angle corresponding to the object's posture determine the robot arm driving information, and finally the robot arm grasps the object.

[0142] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the robot arm control method of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0143] This application also provides a robot arm control device, please refer to Figure 8 , Figure 8 This is a schematic diagram of the module structure of the robot arm control device of the embodiment of the present application. In this embodiment, the robot arm control device includes:

[0144] The image detection module 801 is used to perform part target detection on the material transportation image to obtain a target part image;

[0145] A pose fitting module 802 is used to perform a preset linear pose fitting on the target part image to obtain part pose information;

[0146] An inverse kinematics analysis module 803 is used to perform inverse kinematics analysis based on the part position information and the part posture information corresponding to the material transport image to obtain the robot arm motion information;

[0147] The transport control module 804 is used to control the robot arm to transport parts based on the robot arm motion information.

[0148] As an implementable method, in this embodiment, the posture fitting module 802 is also used to perform a preset linear fitting on the target part image to obtain target main axis information;

[0149] The pose fitting module 802 is further used to determine the target segmentation center of gravity distance according to the center point information corresponding to the target main axis information;

[0150] The pose fitting module 802 is further used to determine the pose information of the part based on the target segmentation center of gravity distance.

[0151] As an implementable method, in this embodiment, the posture fitting module 802 is also used to segment the target part image and detect the center of gravity to obtain the target center of gravity coordinates;

[0152] The posture fitting module 802 is also used to perform linear fitting on the target center of gravity coordinates through the least squares method to obtain target main axis information.

[0153] As an implementable method, in this embodiment, the posture fitting module 802 is further used to perform edge fitting on the target main axis information to obtain the target edge center of gravity point;

[0154] The pose fitting module 802 is further used to determine the target segmentation center of gravity distance according to the center point information corresponding to the target edge center of gravity point and the target main axis information.

[0155] As an implementable method, in this embodiment, the posture fitting module 802 is further used to determine the short axis direction corresponding to the target part according to the target segmentation center of gravity distance;

[0156] The posture fitting module 802 is further used to determine the target rotation angle based on the angle between the straight line equation corresponding to the short axis direction and the horizontal axis;

[0157] The pose fitting module 802 is further used to determine the pose information of the part according to the target rotation angle.

[0158] As an implementable method, in this embodiment, the image detection module 801 is also used to perform part target detection on the material transportation image through a pre-trained YOLOv5 network to obtain an initial part image;

[0159] The image detection module 801 is also used to extract edge contours of the initial part image to obtain a target part image.

[0160] As an implementable method, in this embodiment, the image detection module 801 is further used to perform Gaussian Laplace edge detection on the initial part image to obtain target edge information;

[0161] The image detection module 801 is further used to perform a pixel grayscale traversal on the initial part image according to the target edge information, and to take pixel points that meet a preset edge grayscale standard as target edge points;

[0162] The image detection module 801 is further used to perform image segmentation on the initial part image based on the target edge points to obtain a target part image.

[0163] The robot arm control device provided by the present application adopts the robot arm control method in the above embodiment, which can solve the technical problem of robot arm control. Compared with the prior art, the beneficial effects of the robot arm control device provided by the present application are the same as the beneficial effects of the robot arm control method provided by the above embodiment, and other technical features in the robot arm control device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0164] The present application provides a robotic arm control device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the robotic arm control method in the above-mentioned embodiment one.

[0165] Reference below Fig. 9 , which shows a schematic diagram of the structure of a mechanical arm control device suitable for implementing the embodiment of the present application. The mechanical arm control device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9The robot arm control device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0166] like Fig. 9 As shown, the robot arm control device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the robot arm control device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the robot control device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a robot control device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0167] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a robot arm control program product, which includes a robot arm control program carried on a computer-readable medium, and the robot arm control program contains program code for executing the method shown in the flowchart. In such an embodiment, the robot arm control program can be downloaded and installed from the network through a communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the robot arm control program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0168] The robot arm control device provided by the present application adopts the robot arm control method in the above embodiment, which can solve the technical problem of robot arm control. Compared with the prior art, the beneficial effects of the robot arm control device provided by the present application are the same as the beneficial effects of the robot arm control method provided by the above embodiment, and the other technical features in the robot arm control device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0169] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0170] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0171] The present application provides a storage medium having computer-readable program instructions (ie, a robotic arm control program) stored thereon, wherein the computer-readable program instructions are used to execute the robotic arm control method in the above-mentioned embodiment.

[0172] The storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: RandomAccess Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0173] The above storage medium may be included in the robot arm control device; or may exist independently without being assembled into the robot arm control device.

[0174] The storage medium carries one or more programs. When the one or more programs are executed by the robot arm control device, the robot arm control device performs: robot arm control.

[0175] The robotic arm control program code for performing the operations of the present application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0176] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and robot control program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0177] The modules involved in the embodiments of the present application may be implemented by software or hardware, wherein the name of the module does not limit the unit itself in some cases.

[0178] The readable storage medium provided by the present application is a storage medium, which stores computer-readable program instructions (i.e., a robot arm control program) for executing the above-mentioned robot arm control method, and can solve the technical problem of robot arm control. Compared with the prior art, the beneficial effects of the storage medium provided by the present application are the same as the beneficial effects of the robot arm control method provided by the above-mentioned embodiment, and will not be repeated here.

[0179] The above are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A robot arm control method, characterized in that: The robot arm control method comprises: Perform part target detection on the material transportation image to obtain the target part image; Performing a preset linear pose fitting on the target part image to obtain part pose information; Perform inverse kinematics solution based on the part position information corresponding to the material transportation image and the part posture information to obtain the robot arm motion information; The robot arm is controlled to transport parts based on the robot arm motion information.

2. The robot arm control method according to claim 1, characterized in that: The step of performing preset linear pose fitting on the target part image to obtain part pose information comprises: Performing a preset linear fitting on the target part image to obtain target principal axis information; Determine the target segmentation center of gravity distance according to the center point information corresponding to the target main axis information; The part pose information is determined based on the target segmentation center of gravity distance.

3. The robot arm control method according to claim 2, characterized in that: The step of performing a preset linear fitting on the target part image to obtain target spindle information includes: Segmenting the target part image and detecting the center of gravity to obtain the target center of gravity coordinates; The target centroid coordinates are linearly fitted by the least square method to obtain the target principal axis information.

4. The robot arm control method according to claim 2, characterized in that: The step of determining the target segmentation center of gravity distance according to the center point information corresponding to the target main axis information includes: Performing edge fitting on the target principal axis information to obtain the target edge center of gravity; The target segmentation center of gravity distance is determined according to the center point information corresponding to the target edge center of gravity point and the target main axis information.

5. The robot arm control method according to claim 2, characterized in that: The step of determining part pose information based on the target segmentation center of gravity distance comprises: Determine the short axis direction corresponding to the target part according to the target segmentation center of gravity distance; Determine the target rotation angle based on the angle between the straight line equation corresponding to the short axis direction and the horizontal axis; The part position and posture information is determined according to the target rotation angle.

6. The robot arm control method according to claim 1, characterized in that: The step of performing part target detection on the material transportation image to obtain the target part image comprises: Use the pre-trained YOLOv5 network to detect parts in material transportation images and obtain initial part images. The edge contour of the initial part image is extracted to obtain a target part image.

7. The robot arm control method according to claim 6, characterized in that: The step of extracting edge contours from the initial part image to obtain a target part image comprises: Performing Laplacian Gaussian edge detection on the initial part image to obtain target edge information; Performing pixel grayscale traversal on the initial part image according to the target edge information, and taking pixel points that meet the preset edge grayscale standard as target edge points; The initial part image is segmented based on the target edge points to obtain a target part image.

8. A robot arm control device, characterized in that: The mechanical arm control device comprises: An image detection module is used to detect target parts in material transportation images and obtain target part images; A posture fitting module, used to perform a preset linear posture fitting on the target part image to obtain part posture information; An inverse kinematics analysis module, used to perform inverse kinematics analysis based on the part position information corresponding to the material transport image and the part posture information to obtain the robot arm motion information; The transport control module is used to control the robot arm to transport parts based on the movement information of the robot arm.

9. A robot arm control device, characterized in that: The device comprises: a memory, a processor, and a robot arm control program stored in the memory and executable on the processor, wherein the robot arm control program is configured to implement the steps of the robot arm control method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a robot arm control program, and when the robot arm control program is executed by the processor, the steps of the robot arm control method according to any one of claims 1 to 7 are implemented.

Images