Robotic arm grasping method, system, device, electronic device and storage medium

Abstract

The present application relates to a robotic arm grasping method, system, device, electronic device, and storage medium. The method comprises: determining a first actual coordinate of a target object in a robotic arm coordinate system based on a first pixel coordinate of the target object in a corresponding object photograph; determining a second actual coordinate of the robotic arm in the robotic arm coordinate system when the robotic arm grasps the target object based on the first actual coordinate; determining an attitude angle of the robotic arm corresponding to the second actual coordinate; and controlling the robotic arm to grasp the target object based on the second actual coordinate and the attitude angle. This method can improve the efficiency of robotic arm grasping.

Description

Technical Field

[0001] The present application relates to the field of robotic arm technology, and in particular to a robotic arm grasping method, system, device, electronic device and storage medium. Background Art

[0002] With the development of robotic arm technology, industrial robotic arms have emerged for use in assembly line operations. Before a robotic arm can perform assembly line operations, a grasping plan must be pre-designed based on the model, size, and shape of the product to be processed, as well as the size and shape of the end-of-arm tooling. This allows the robotic arm to perform the assembly line operation according to the designed grasping plan.

[0003] However, using the above method, when the assembly line operation needs to be changed, it is usually necessary to redesign the grasping plan for the robot arm. The plan formulation process takes a lot of time, which is not conducive to improving production efficiency.

[0004] Therefore, the current robotic arm grasping technology has the problem of low production efficiency. Summary of the Invention

[0005] Based on this, it is necessary to provide a robotic arm method, system, device, electronic device and storage medium that can improve production efficiency in response to the above technical problems.

[0006] In a first aspect, the present application provides a robotic arm grasping method. The method comprises:

[0007] Determining a first actual coordinate of the target object in the robotic arm coordinate system according to a first pixel coordinate of the target object in the corresponding object photo;

[0008] determining, based on the first actual coordinate, a second actual coordinate of the robotic arm in the robotic arm coordinate system when the robotic arm grasps the target object;

[0009] Determining the attitude angle of the robotic arm corresponding to the second actual coordinate;

[0010] The robotic arm is controlled to grasp the target object according to the second actual coordinate and the posture angle.

[0011] In a second aspect, the present application also provides a robotic arm grasping system. The system includes a camera, a controller, and a robotic arm;

[0012] The camera is used to photograph the target object to obtain an object photo corresponding to the target object;

[0013] The controller is configured to determine a first actual coordinate of the target object in the robotic arm coordinate system based on a first pixel coordinate of the target object in the object photograph, determine a second actual coordinate of the robotic arm in the robotic arm coordinate system when the robotic arm grasps the target object based on the first actual coordinate, determine a posture angle of the robotic arm corresponding to the second actual coordinate, and control the robotic arm to grasp the target object based on the second actual coordinate and the posture angle;

[0014] The robotic arm is used to grasp the target object according to the second actual coordinate and the posture angle.

[0015] In a third aspect, the present application further provides a robotic arm grasping device. The device comprises:

[0016] A coordinate conversion module, configured to determine a first actual coordinate of the target object in the robotic arm coordinate system according to a first pixel coordinate of the target object in the corresponding object photo;

[0017] a coordinate determination module, configured to determine, based on the first actual coordinate, a second actual coordinate of the robotic arm in the robotic arm coordinate system when the robotic arm grasps the target object;

[0018] An angle determination module, configured to determine a posture angle of the robotic arm corresponding to the second actual coordinate;

[0019] A grasping control module is used to control the robotic arm to grasp the target object according to the second actual coordinate and the posture angle.

[0020] In a fourth aspect, the present application further provides an electronic device comprising: a memory, one or more processors;

[0021] The memory is used to store one or more programs;

[0022] When the one or more programs are executed by the one or more processors, the one or more processors implement the following steps:

[0023] Determining a first actual coordinate of the target object in the robotic arm coordinate system according to a first pixel coordinate of the target object in the corresponding object photo;

[0024] determining, based on the first actual coordinate, a second actual coordinate of the robotic arm in the robotic arm coordinate system when the robotic arm grasps the target object;

[0025] Determining the attitude angle of the robotic arm corresponding to the second actual coordinate;

[0026] The robotic arm is controlled to grasp the target object according to the second actual coordinate and the posture angle.

[0027] In a fifth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0028] Determining a first actual coordinate of the target object in the robotic arm coordinate system according to a first pixel coordinate of the target object in the corresponding object photo;

[0029] determining, based on the first actual coordinate, a second actual coordinate of the robotic arm in the robotic arm coordinate system when the robotic arm grasps the target object;

[0030] Determining the attitude angle of the robotic arm corresponding to the second actual coordinate;

[0031] The robotic arm is controlled to grasp the target object according to the second actual coordinate and the posture angle.

[0032] The above-mentioned robot arm grasping method, system, device, electronic device and storage medium determine the first actual coordinates of the target object in the robot arm coordinate system according to the first pixel coordinates of the target object in the corresponding object photo, determine the second actual coordinates of the robot arm in the robot arm coordinate system when the robot arm grasps the target object according to the first actual coordinates, determine the posture angle of the robot arm corresponding to the second actual coordinates, and control the robot arm to grasp the target object according to the second actual coordinates and the posture angle; the pixel coordinates of the target object in the photo can be converted into actual coordinates, and the actual coordinates of the target object can be determined according to the actual coordinates of the target object. The actual coordinates of the robot arm when it grabs the target object to the specified position on the assembly line, or grabs the target object from the specified position on the assembly line, and by determining the posture angle corresponding to the actual coordinates of the robot arm, the posture of the robot arm when it grabs the target object to the specified position on the assembly line, or grabs the target object from the specified position on the assembly line is obtained. This makes it possible to control the robot arm to grab the target object based on the actual coordinates and posture of the robot arm when grabbing the target object. Therefore, it is only necessary to set up a camera on the production line and use the camera to shoot the target object to quickly formulate a robot arm grabbing plan, thereby improving the efficiency of the robot arm grabbing.

[0033] Moreover, since the robot arm grasping plan can be quickly formulated based on the photos taken, the line change time is reduced, and it is also convenient to implement mixed-line production on the production line to meet the diverse production needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is an application environment diagram of a robotic arm grasping method;

[0035] Figure 2 This is an application environment diagram of a robotic arm grasping method provided in Example 1 of the present application;

[0036] Figure 3 This is a flow chart of a robotic arm grasping method provided in Example 1 of the present application;

[0037] Figure 4 This is a schematic diagram of the positional relationship between an object and a robotic arm on an assembly line provided in Example 1 of the present application;

[0038] Figure 5 This is a flow chart of a robotic arm grasping method provided in Example 2 of the present application;

[0039] Figure 6 is a schematic diagram of the teaching point provided in Example 2 of the present application;

[0040] Figure 7 This is a partial enlarged view of the teaching point provided in Example 2 of the present application;

[0041] Figure 8 is a schematic diagram of another teaching point provided in Example 2 of the present application;

[0042] Figure 9 This is a flowchart of the teaching steps provided in Example 2 of this application;

[0043] Figure 10 This is a flow chart of the production steps provided in Example 2 of the present application;

[0044] Figure 11 This is a schematic structural diagram of a robotic arm grasping system provided in Example 3 of the present application;

[0045] Figure 12 This is a structural diagram of a robotic arm grasping device provided in Example 4 of the present application;

[0046] Figure 13 This is a schematic diagram of the structure of an electronic device provided in Example 5 of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0048] Figure 1The present invention provides an application environment of the current robotic arm grasping method, which mainly includes: a target object 101, a robotic arm 102 and a controller 108, wherein the target object 101 is the object of the assembly line operation, the controller 108 can be a terminal or a server, and a flange device 105 is provided at the end of the robotic arm 102, and a tool 104 is installed on the flange device 105. The robotic arm 102 can use the tool 104 to grasp the target object 101 to a specified position on the assembly line 103, or grasp the target object 101 from a specified position on the assembly line 103. Figure 1 The robotic arm 102 communicates with the controller 108 , and the controller 108 can send control instructions to the robotic arm 102 . The robotic arm 102 performs a grasping operation on the target object 101 placed on the assembly line 103 according to the received control instructions.

[0049] Before an assembly line begins, it's necessary to comprehensively consider the target object's model, size, and shape, as well as the tool's size and shape, and then develop a grasping plan within the controller based on the line's operational requirements. This plan then allows the robotic arm to execute the assembly line operation according to the configured grasping plan. However, when lines need to be changed or mixed, the grasping plan often needs to be re-developed, and this process can involve multiple manual adjustments, resulting in low efficiency.

[0050] In view of this, the embodiment of the present application provides a robotic arm grasping method, which can be applied to Figure 2 In the application environment shown, a camera 106 is provided above the assembly line 103 for photographing the target object 101 and the robotic arm 102 on the assembly line 103. The robotic arm 102 and the camera 106 both communicate with the controller 108. For the robotic arm grasping method, a robotic arm coordinate system and a camera coordinate system can be set. The robotic arm coordinate system can be a coordinate system of an actual three-dimensional space, which is used to determine the actual coordinates of the robotic arm 102, the flange device 105, the tool 104, and the target object 101 in the three-dimensional space. The camera coordinate system can be a coordinate system of a photo taken by the camera 106, which is used to determine the pixel coordinates of the robotic arm 102, the flange device 105, the tool 104, and the target object 101 in the photo.

[0051] Example 1

[0052] Figure 3 This is a flow chart of a robotic arm grasping method provided in Example 1 of this application. Figure 3 , taking the application in the controller as an example, the robot arm grasping method of the first embodiment of the present application specifically includes:

[0053] Step S210 : determining a first actual coordinate of the target object in the robot arm coordinate system according to a first pixel coordinate of the target object in the corresponding object photo.

[0054] The object photo may be a photo obtained by photographing the target object with a camera.

[0055] The first pixel coordinates may be pixel coordinates of the target object in the camera coordinate system.

[0056] The first actual coordinate may be the actual coordinate of the target object in the robotic arm coordinate system.

[0057] In a specific implementation, a camera can be used to photograph the target object on the assembly line to obtain a photo of the object. The camera sends the photo of the object to the controller. The controller can pre-store a mapping relationship between the camera coordinate system and the robotic arm coordinate system. The controller can transform the first pixel coordinate of the target object in the camera coordinate system to the first actual coordinate of the target object in the robotic arm coordinate system based on the mapping relationship.

[0058] In actual applications, a sensor can be set on the assembly line. When the sensor detects the target object, it sends a shooting instruction to the controller. When receiving the shooting instruction, the controller can control the camera to shoot the assembly line to obtain a photo of the object. The camera can send the photographed object photo to the controller. The controller identifies the received object photo and takes the center point of the identified target object as the first pixel coordinate. Through the pre-stored mapping relationship between the camera coordinate system and the robotic arm coordinate system, the first pixel coordinate is transformed into the first actual coordinate to obtain the actual coordinate of the center point of the target object in the robotic arm coordinate system.

[0059] It should be noted that the mapping relationship between the camera coordinate system and the robotic arm coordinate system can be determined by teaching the robotic arm. A teaching point can be pre-set on the assembly line, and the robotic arm can be controlled to grasp a reference object at the teaching point. When the robotic arm grasps the reference object and reaches the teaching point, the camera is controlled to capture the reference object, obtaining a photo of the reference object. The photo of the reference object can include the flange device, tools, and the reference object. The camera then sends the photo of the reference object to the controller. The controller can obtain the actual coordinates of the robotic arm when the robotic arm grasps the reference object and reaches the teaching point. It can also identify the pixel coordinates of the reference object in the reference object photo and determine the mapping relationship between the actual coordinates of the robotic arm and the pixel coordinates of the reference object as the mapping relationship between the camera coordinate system and the robotic arm coordinate system. The controller can also determine the distance between the robotic arm and the reference object when the robotic arm grasps the reference object and reaches the teaching point. This distance can be used to calibrate the mapping relationship between the camera coordinate system and the robotic arm coordinate system, resulting in a calibrated mapping relationship between the camera coordinate system and the robotic arm coordinate system.

[0060] Step S220 : determining, based on the first actual coordinate, a second actual coordinate of the robotic arm in the robotic arm coordinate system when the robotic arm grasps the target object.

[0061] The second actual coordinate may be the actual coordinate in the robotic arm coordinate system when the robotic arm grasps the target object.

[0062] In a specific implementation, the deviation and angle between the robotic arm and the target object in the robotic arm coordinate system can be determined in advance. After determining the first actual coordinate of the target object in the robotic arm coordinate system according to the first pixel coordinate in step S210, the second actual coordinate of the robotic arm in the robotic arm coordinate system can be determined based on the deviation and angle between the robotic arm and the target object in the robotic arm coordinate system when the robotic arm grabs the target object to a specified position on the assembly line, or grabs the target object from a specified position on the assembly line.

[0063] Figure 4 This is a schematic diagram of the positional relationship between the object and the robotic arm on the assembly line provided by the first embodiment of the present application. Figure 4 In the robot coordinate system, A represents the center of the robot flange assembly, and B represents the center of the target object. Line AB can be considered a virtual tool, with a length of L and an angle θ between the virtual tool and the horizontal line AC. Therefore, the lateral offset (offsetX) and longitudinal offset (offsetY) of A relative to B are offsetX = Lcosθ and offsetY = Lsinθ, respectively. After determining the first actual coordinates of the target object's center (x'0, y'0, 1), the actual coordinates of the robot flange assembly's center (x'0 - Lcosθ, y'0 - Lsinθ, 1) can be obtained based on the virtual tool's length L and angle θ. These coordinates are used as the second actual coordinates.

[0064] Step S230: Determine the posture angle of the robotic arm corresponding to the second actual coordinate.

[0065] The attitude angle may be the heading angle, pitch angle, and roll angle of the robotic arm in the robotic arm coordinate system.

[0066] In a specific implementation, after obtaining the second actual coordinate, the attitude angle when the robot arm grabs the target object to a specified position on the assembly line or grabs the target object from a specified position on the assembly line can be determined based on the second actual coordinate.

[0067] In practical applications, the robot arm attitude angle can be set to (R x , R y , R z ), where the component R x 、R y 、R zRepresents heading angle, pitch angle, and roll angle respectively. According to the nature of the robot arm operation, R x and R y is known and fixed, so only the component R is required z The actual coordinates of the target object in the robot coordinate system can be obtained, the actual coordinates of the target object in the robot coordinate system are connected with the origin of the robot coordinate system, and the angle between the connecting line and the X-axis of the robot coordinate system is determined. The angle between the connecting line and the X-axis of the robot coordinate system is added to the virtual tool angle θ to obtain the component R z .

[0068] Step S240: Control the robotic arm to grasp the target object according to the second actual coordinate and the posture angle.

[0069] In the specific implementation, after determining the second actual coordinate and attitude angle, if the robotic arm is required to grab the target object to the specified position on the assembly line, since the starting coordinate and starting attitude angle of the robotic arm are known, the path of the robotic arm can be planned and the movement of the robotic arm can be controlled based on the starting coordinate, starting attitude angle, the second actual coordinate and the attitude angle corresponding to the second actual coordinate; if the robotic arm is required to grab the target object from the specified position on the assembly line, the robotic arm can be directly controlled to grab the target object based on the second actual coordinate and the attitude angle corresponding to the second actual coordinate.

[0070] The above-mentioned robotic arm grasping method determines the first actual coordinates of the target object in the robotic arm coordinate system according to the first pixel coordinates of the target object in the corresponding object photo, determines the second actual coordinates of the robotic arm in the robotic arm coordinate system when the robotic arm grasps the target object according to the first actual coordinates, determines the attitude angle of the robotic arm corresponding to the second actual coordinates, and controls the robotic arm to grasp the target object according to the second actual coordinates and the attitude angle; the pixel coordinates of the target object in the photo can be converted into actual coordinates, and the actual coordinates when the robotic arm grasps the target object to a specified position on the assembly line or grasps the target object from a specified position on the assembly line can be determined according to the actual coordinates of the target object, and the attitude angle corresponding to the actual coordinates of the robotic arm can be obtained when the robotic arm grasps the target object to a specified position on the assembly line or grasps the target object from a specified position on the assembly line, so that the robotic arm can be controlled to grasp the target object according to the actual coordinates and attitude of the robotic arm grasping the target object, thereby only needing to set up a camera on the production line and photograph the target object with the camera to quickly formulate a robotic arm grasping plan, thereby improving the efficiency of robotic arm grasping.

[0071] Moreover, since the robot arm grasping plan can be quickly formulated based on the photos taken, the line change time is reduced, and it is also convenient to implement mixed-line production on the production line to meet the diverse production needs.

[0072] Example 2

[0073] Figure 5 This is a flow chart of a robotic arm grasping method provided in Example 2 of this application. Figure 5 , taking the application in the controller as an example, the robot arm grasping method of the second embodiment of the present application specifically includes:

[0074] Step S310 , determining a first mapping relationship between a camera coordinate system where the object photo is located and a robotic arm coordinate system where the target object is located.

[0075] The first mapping relationship may be a mapping relationship between the pixel coordinates of the object in the camera coordinate system and the actual coordinates of the robotic arm in the robotic arm coordinate system.

[0076] In a specific implementation, at least one teaching point can be set on the assembly line, and the robotic arm can be controlled to grab the reference object to each teaching point respectively. When the reference object arrives at each teaching point, a photo of the reference object is taken, and the pixel coordinates of the reference object in the reference object photo are obtained. The actual coordinates of the robotic arm when the robotic arm grabs the reference object to each teaching point can also be obtained, and the mapping relationship between the pixel coordinates of the reference object and the actual coordinates of the robotic arm is determined as the first mapping relationship.

[0077] The reference object may be a physical object used to measure the first mapping relationship.

[0078] The teaching point may be a point on the production line where a reference object is placed in advance.

[0079] Figure 6 This is a schematic diagram of the teaching points provided in Example 2 of the present application. Figure 7 This is a partially enlarged view of the teaching point provided in Example 2 of the present application. Figure 8 This is a schematic diagram of another teaching point provided in Example 2 of this application. Figure 6 and Figure 7 , you can set the standard storage point P1 and teaching points P2 to P10 on the assembly line. In the robot coordinates, P2 to P10 correspond to different actual coordinates (x' i , y' i , 1), i = 2, ..., 10, at the same time, P2 ~ P10 correspond to the same attitude angle (R x , R y , R z ).according to Figure 8 , the teaching points P11 and P12 can be obtained by rotating P2. The actual coordinates of P11 and P12 in the robot coordinate system are the same as P2, both (x'2, y'2, 1). The R in the attitude angle x and Ry Same as P2, only R z Different from P2.

[0080] Optionally, the step S310 includes:

[0081] Obtain the second pixel coordinates of the reference object in the camera coordinate system, and obtain the third actual coordinates of the robotic arm in the robotic arm coordinate system when the robotic arm grasps the reference object; and determine the mapping relationship between the second pixel coordinates and the third actual coordinates as the first mapping relationship between the camera coordinate system and the robotic arm coordinate system.

[0082] The second pixel coordinates may be pixel coordinates of the reference object in the camera coordinate system.

[0083] The third actual coordinate may be the actual coordinate in the robotic arm coordinate system when the robotic arm grasps the reference object.

[0084] In a specific implementation, at least one teaching point can be set on the assembly line, and the robotic arm can be controlled to grab the reference object to each teaching point respectively. When the reference object arrives at each teaching point, a photo of the reference object is taken, and the second pixel coordinates of the reference object in the reference object photo are obtained. The third actual coordinates of the robotic arm in the robotic arm coordinate system when the robotic arm grabs the reference object to each teaching point can also be obtained, and the mapping relationship between the second pixel coordinate of the reference object and the third actual coordinate of the robotic arm is determined as the first mapping relationship.

[0085] refer to Figure 6 and Figure 7 , the robot arm can be controlled to grab the reference object from P1 to the teaching points P2 to P10, and take a photo when the reference object reaches each teaching point, and identify the second pixel coordinate (x i ,y i ), i = 2, ..., 10, and the third actual coordinate (x') of the center point of the flange device on the robot arm in the robot arm coordinate system can also be recorded when the reference object reaches each teaching point. i , y' i , 1), i = 2, ..., 10, according to the second pixel coordinate and the third actual coordinate, the mapping relationship M1 can be obtained, and the specific formula can be

[0086]

[0087] in, is the matrix corresponding to the second pixel coordinate, Represents the mapping relationship M1, is the matrix corresponding to the third actual coordinate.

[0088] According to the technical solution of the embodiment of the present application, the second pixel coordinates of the reference object in the camera coordinate system are obtained, and the third actual coordinates of the robotic arm in the robotic arm coordinate system when the robotic arm grasps the reference object are obtained. The mapping relationship between the second pixel coordinates and the third actual coordinates is determined as the first mapping relationship between the camera coordinate system and the robotic arm coordinate system. The mapping relationship between the camera coordinate system and the robotic arm coordinate system can be quickly determined, the time for formulating the robotic arm grasping plan can be shortened, and the efficiency of the robotic arm grasping can be improved.

[0089] Step S320: Correct the first mapping relationship to obtain a second mapping relationship between the camera coordinate system and the robotic arm coordinate system.

[0090] The second mapping relationship may be a mapping relationship between the pixel coordinates of the object in the camera coordinate system and the actual coordinates of the object in the robot arm coordinate system.

[0091] In the specific implementation, the first mapping relationship is calculated based on the pixel coordinates of the reference object in the camera coordinate system and the actual coordinates of the robotic arm in the robotic arm coordinate system. The reference objects are the reference object and the robotic arm respectively. Since there is a certain distance between the reference object and the robotic arm, it is easy to cause the first mapping relationship to be determined inaccurately. Therefore, the first mapping relationship needs to be corrected, and the actual coordinates of the reference object are determined based on the actual coordinates of the robotic arm, and then the second mapping relationship is calculated based on the pixel coordinates of the reference object in the camera coordinate system and the actual coordinates in the robotic arm coordinate system.

[0092] Optionally, the step S320 includes:

[0093] Determine the actual deviation between the robotic arm and the reference object in the robotic arm coordinate system; correct the third actual coordinate of the robotic arm based on the actual deviation to obtain the fourth actual coordinate of the reference object in the robotic arm coordinate system; and determine the mapping relationship between the second pixel coordinate and the fourth actual coordinate as the second mapping relationship between the camera coordinate system and the robotic arm coordinate system.

[0094] The actual deviation may be a deviation in the robot arm coordinate system.

[0095] The fourth actual coordinate may be the actual coordinate of the reference object in the robotic arm coordinate system.

[0096] In a specific implementation, the actual deviation between the robotic arm and the reference object when the robotic arm grasps the reference object can be determined in advance. After obtaining the third actual coordinate of the robotic arm in the robotic arm coordinate system when the robotic arm grasps the reference object to each teaching point, the third actual coordinate can be corrected according to the actual deviation to obtain the fourth actual coordinate of the reference object in the robotic arm coordinate system when the robotic arm grasps the reference object to each teaching point. The mapping relationship between the second pixel coordinate and the fourth actual coordinate of the reference object is determined as the second mapping relationship.

[0097] refer to Figure 4 , we can determine the actual deviation between the center point of the manipulator and the center point of the reference object as offsetX=Lcosθ,offsetY=Lsinθ, and then get the third actual coordinate of the manipulator (x' i , y' i , 1), the third actual coordinate can be corrected according to the actual deviation to obtain the fourth actual coordinate (x' i +offsetX,y' i + offsetY, 1), according to the second pixel coordinate and the fourth actual coordinate, the mapping relationship M2 can be obtained. The specific formula can be

[0098]

[0099] in, is the matrix corresponding to the second pixel coordinate, Represents the mapping relationship M2, is the matrix corresponding to the fourth actual coordinate.

[0100] According to the technical solution of the embodiment of the present application, the actual deviation between the robotic arm and the reference object in the robotic arm coordinate system is determined. Based on the actual deviation, the third actual coordinate of the robotic arm is corrected to obtain the fourth actual coordinate of the reference object in the robotic arm coordinate system; the mapping relationship between the second pixel coordinate and the fourth actual coordinate is determined as the second mapping relationship between the camera coordinate system and the robotic arm coordinate system, so that the acquired fourth actual coordinate and the second pixel coordinate are both relative to the reference object. The mapping relationship between the camera coordinate system and the robotic arm coordinate system is determined based on the second pixel coordinate and the fourth actual coordinate, which can improve the accuracy of the determination of the mapping relationship.

[0101] Optionally, determining an actual deviation between the robotic arm and the reference object in the robotic arm coordinate system includes:

[0102] Determine the third pixel coordinate of the center point of the reference object in the camera coordinate system, and determine the fourth pixel coordinate of the center point of the robotic arm in the camera coordinate system; determine the fifth actual coordinate corresponding to the third pixel coordinate and the sixth actual coordinate corresponding to the fourth pixel coordinate based on the first mapping relationship; and determine the deviation between the fifth actual coordinate and the sixth actual coordinate as the actual deviation between the robotic arm and the reference object.

[0103] The third pixel coordinates may be the pixel coordinates of the center point of the reference object in the camera coordinate system.

[0104] The fourth pixel coordinate may be the pixel coordinate of the center point of the flange device on the robotic arm in the camera coordinate system.

[0105] The fifth actual coordinate may be the actual coordinate of the center point of the reference object in the robotic arm coordinate system.

[0106] The sixth actual coordinate may be the actual coordinate of the center point of the flange device on the robotic arm in the robotic arm coordinate system.

[0107] In a specific implementation, the center point of the reference object can be identified in the reference object photo, and the third pixel coordinate of the center point of the reference object in the reference object photo can be determined. The center point of the robotic arm flange device can also be identified in the reference object photo, and the fourth pixel coordinate of the center point of the flange device in the reference object photo can be determined. According to the first mapping relationship, the third pixel coordinate and the fourth pixel coordinate are mapped respectively to obtain the fifth actual coordinate and the sixth actual coordinate. Since the fifth actual coordinate is the actual coordinate of the center point of the reference object in the robotic arm coordinate system, and the sixth actual coordinate is the actual coordinate of the center point of the flange device in the robotic arm coordinate system, the deviation between the fifth actual coordinate and the sixth actual coordinate can be determined as the actual deviation between the robotic arm and the reference object.

[0108] In practical applications, the robot arm can be controlled to grab the reference object to P2. The pixel coordinates of the center point of the reference object are (x2, y2). The pixel coordinates of the center point of the robot arm flange device at this time can also be determined (x 13 ,y 13 ), use M1 to transform the pixel coordinates to obtain the actual coordinates of the center of the reference object (x'2, y'2, 1) and the actual coordinates of the center of the robot flange device (x' 13 , y' 13 , 1), calculate (x'2, y'2, 1) and (x' 13 , y' 13 , 1) the distance L between them, and the angle θ between the line connecting the two and the horizontal line, refer to Figure 4, we can get the lateral deviation offsetX=Lcosθ and the longitudinal deviation offsetY=Lsinθ between the robot arm and the reference object.

[0109] According to the technical solution of the embodiment of the present application, the third pixel coordinate of the center point of the reference object in the camera coordinate system is determined, and the fourth pixel coordinate of the center point of the robotic arm in the camera coordinate system is determined. According to the first mapping relationship, the fifth actual coordinate corresponding to the third pixel coordinate and the sixth actual coordinate corresponding to the fourth pixel coordinate are determined. The deviation between the fifth actual coordinate and the sixth actual coordinate is determined as the actual deviation between the robotic arm and the reference object. This can facilitate the correction of the third actual coordinate of the robotic arm according to the actual deviation, thereby improving the accuracy of the mapping relationship between the camera coordinate system and the robotic arm coordinate system.

[0110] Optionally, the center point of the robotic arm includes the center point of a flange device of the robotic arm; and determining the fourth pixel coordinate of the center point of the robotic arm in the camera coordinate system includes:

[0111] Obtain at least three tool pixel coordinates of the tool installed at the center point of the flange device in the camera coordinate system; each tool pixel coordinate corresponds to a different posture angle of the tool; and determine the center of the circle corresponding to the at least three tool pixel coordinates as the fourth pixel coordinate of the center point of the robotic arm in the camera coordinate system.

[0112] The tool pixel coordinates can be the pixel coordinates of the tool end in the camera coordinate system. The tool is mounted at the center of the flange assembly of the robot arm. The end of the tool connected to the center of the flange assembly is called the front end, and the end away from the flange assembly is called the end end.

[0113] In a specific implementation, a tool can be mounted at the center of a flange assembly on a robotic arm. With the flange assembly stationary, the front end of the tool is fixed while the rear end rotates, yielding at least three rotational postures of the tool. A camera is then used to capture each rotational posture, obtaining at least three photos of the tool. The pixel coordinates of the tool rear end in each photo are used as the tool pixel coordinates, yielding at least three tool pixel coordinates, each corresponding to a different rotational angle of the tool. Based on the principle of three points defining a circle, the center of the circle corresponding to the at least three tool pixel coordinates can be determined as the fourth pixel coordinate of the robotic arm's center point in the camera coordinate system.

[0114] refer to Figure 8 , the tool can be installed at the center point of the robot arm flange device, the front end of the tool is fixed, and the end can rotate. The tool with the end rotated to the P2 position is photographed for the first time, and the pixel coordinates of the tool are determined from the photographed photos as (x 2tool ,y 2tool), the tool with the end rotated to the P11 position is photographed for the second time, and the pixel coordinates of the tool are determined from the photographed photos as (x 11tool ,y 11tool ), the tool at the end rotated to the P12 position is photographed for the third time, and the pixel coordinates of the tool are determined from the photographed photos as (x 12tool ,y 12tool ), since three points can determine a circle, according to (x 2tool ,y 2tool )、(x 11tool ,y 11tool )、(x 12tool ,y 12tool ) can determine a circle, and the center of the circle can be used as the pixel coordinate of the center point of the robot arm flange device, that is, the fourth pixel coordinate of the center point of the robot arm in the camera coordinate system.

[0115] According to the technical solution of the embodiment of the present application, at least three tool pixel coordinates of the tool installed at the center point of the flange device in the camera coordinate system are obtained; each tool pixel coordinate corresponds to a different posture angle of the tool; the center of the circle corresponding to the at least three tool pixel coordinates is determined as the fourth pixel coordinate of the center point of the robotic arm in the camera coordinate system, which can quickly determine the pixel coordinates of the center point of the robotic arm, shorten the time for formulating the robotic arm grasping plan, and improve the efficiency of the robotic arm grasping.

[0116] Optionally, the actual deviation includes a lateral deviation and a longitudinal deviation; after determining the deviation between the fifth actual coordinate and the sixth actual coordinate as the actual deviation between the robotic arm and the reference object, the method further includes:

[0117] The distance between the fifth actual coordinate and the sixth actual coordinate is determined as the tool length of the virtual tool; the virtual tool corresponds to the line between the center point of the reference object and the center point of the robotic arm; the tool angle of the virtual tool is determined based on at least two of the lateral deviation, the longitudinal deviation and the tool length; the lateral deviation, the longitudinal deviation, the tool length and the tool angle are used to determine the second actual coordinate based on the first actual coordinate.

[0118] The virtual tool may be a line connecting the center point of the robot arm flange and the center point of the object. The tool length may be the length of the virtual tool. The tool angle may be the angle between the virtual tool and the horizontal line.

[0119] In a specific implementation, after determining the fifth actual coordinate of the object center point and the sixth actual coordinate of the flange device center point, the line connecting the object center point and the flange device center point can be considered as a virtual tool, and the distance between the fifth actual coordinate and the sixth actual coordinate can be determined as the tool length of the virtual tool. The tool length can be measured, and the lateral deviation and longitudinal deviation between the fifth actual coordinate and the sixth actual coordinate can also be measured. Based on at least two of the tool length, the lateral deviation, and the longitudinal deviation, the tool angle of the virtual tool can be determined. After obtaining the lateral deviation, the longitudinal deviation, the tool length, and the tool angle, the second actual coordinate can be determined based on the lateral deviation, the longitudinal deviation, the tool length, and the tool angle.

[0120] refer to Figure 4 The line AB connecting the fifth actual coordinate B and the sixth actual coordinate A can be determined as a virtual tool, with a virtual tool length of L. A lateral offset, offsetX, and a longitudinal offset, offsetY, can also be measured. Based on trigonometric functions, a tool angle of the virtual tool can be determined based on at least two of the lateral offset, longitudinal offset, and tool length. For example, tool angle θ = arctan(offsetY / offsetX). Other methods are not described in detail here. The method for determining the second actual coordinate based on the lateral offset, longitudinal offset, tool length, and tool angle has been described in the previous embodiment and will not be repeated here.

[0121] According to the technical solution of the embodiment of the present application, the distance between the fifth actual coordinate and the sixth actual coordinate is determined as the tool length of the virtual tool; the virtual tool corresponds to the line between the center point of the reference object and the center point of the robotic arm; the tool angle of the virtual tool is determined based on at least two of the lateral deviation, the longitudinal deviation and the tool length; the lateral deviation, the longitudinal deviation, the tool length and the tool angle are used to determine the second actual coordinate based on the first actual coordinate, and the fixed positional relationship between the center point of the reference object and the center point of the robotic arm can be determined. According to the positional relationship, the actual coordinates of the robotic arm are determined from the actual coordinates of the reference object, which facilitates the rapid formulation of the robotic arm grasping plan.

[0122] In step S330 , the first pixel coordinates of the target object in the corresponding object photo are transformed into the first actual coordinates of the target object in the robot arm coordinate system through the second mapping relationship.

[0123] In a specific implementation, a camera can be used to photograph the target object on the assembly line to obtain a photo of the object. The camera sends the photo of the object to the controller. The controller can pre-store a second mapping relationship between the camera coordinate system and the robotic arm coordinate system. The controller can transform the first pixel coordinate of the target object in the camera coordinate system to the first actual coordinate of the target object in the robotic arm coordinate system based on the second mapping relationship.

[0124] Step S340 : determining, based on the first actual coordinate, a second actual coordinate of the robotic arm in the robotic arm coordinate system when the robotic arm grasps the target object.

[0125] In a specific implementation, the deviation and angle between the robotic arm and the target object in the robotic arm coordinate system can be determined in advance. After determining the first actual coordinate of the target object in the robotic arm coordinate system based on the first pixel coordinate, the second actual coordinate of the robotic arm in the robotic arm coordinate system can be determined based on the deviation and angle between the robotic arm and the target object in the robotic arm coordinate system when the robotic arm grabs the target object to a specified position on the assembly line, or grabs the target object from a specified position on the assembly line.

[0126] Step S350: Determine the posture angle of the robotic arm corresponding to the second actual coordinate.

[0127] In a specific implementation, after obtaining the second actual coordinate, the attitude angle when the robot arm grabs the target object to a specified position on the assembly line or grabs the target object from a specified position on the assembly line can be determined based on the second actual coordinate.

[0128] Optionally, the step S350 includes:

[0129] A line connecting the second actual coordinate and the coordinate origin of the robot coordinate system is obtained; an angle between the line and the horizontal coordinate axis of the robot coordinate system is determined; and the angle is summed with the tool angle of the virtual tool to obtain components of the attitude angle.

[0130] In the specific implementation, the robot arm posture angle can be set to (R x , R y , R z ), where the component R x 、R y 、R z Represents heading angle, pitch angle, and roll angle respectively. According to the nature of the robot arm operation, R x and R y is known and fixed, so only the component R is required z The second actual coordinate of the target object in the robot coordinate system can be obtained, the second actual coordinate is connected to the origin of the robot coordinate system, and the angle between the connecting line and the horizontal coordinate axis of the robot coordinate system is determined. The angle between the connecting line and the horizontal coordinate axis of the robot coordinate system is added to the tool angle θ of the virtual tool to obtain the component of the attitude angle R z .

[0131] According to the technical solution of the embodiment of the present application, a line is obtained between the second actual coordinate and the coordinate origin of the robot arm coordinate system; the angle between the line and the horizontal coordinate axis of the robot arm coordinate system is determined; the angle and the tool angle of the virtual tool are summed to obtain the components of the posture angle, which can determine the components of the posture angle of the robot arm when the robot arm grabs the target object to the specified position on the assembly line, or grabs the target object from the specified position on the assembly line, thereby facilitating the rapid formulation of the robot arm grasping plan.

[0132] Step S360: Control the robotic arm to grasp the target object according to the second actual coordinate and the posture angle.

[0133] In the specific implementation, after determining the second actual coordinate and attitude angle, if the robotic arm is required to grab the target object to the specified position on the assembly line, since the starting coordinate and starting attitude angle of the robotic arm are known, the path of the robotic arm can be planned and the movement of the robotic arm can be controlled based on the starting coordinate, starting attitude angle, the second actual coordinate and the attitude angle corresponding to the second actual coordinate; if the robotic arm is required to grab the target object from the specified position on the assembly line, the robotic arm can be directly controlled to grab the target object based on the second actual coordinate and the attitude angle corresponding to the second actual coordinate.

[0134] The above-mentioned robotic arm grasping method determines a first mapping relationship between the camera coordinate system where the object photo is located and the robotic arm coordinate system where the target object is located, corrects the first mapping relationship, and obtains a second mapping relationship between the camera coordinate system and the robotic arm coordinate system. Through the second mapping relationship, the first pixel coordinates of the target object in the corresponding object photo are transformed into the first actual coordinates of the target object in the robotic arm coordinate system. According to the first actual coordinates, the second actual coordinates of the robotic arm in the robotic arm coordinate system when the robotic arm grasps the target object are determined, and the posture angle of the robotic arm corresponding to the second actual coordinates is determined. According to the second actual coordinates and the posture angle, the robotic arm is controlled to grasp the target object; the target object can be grasped. The pixel coordinates of the object in the photo are converted into actual coordinates. According to the actual coordinates of the target object, the actual coordinates when the robot arm grabs the target object to the specified position on the assembly line, or grabs the target object from the specified position on the assembly line are determined, and by determining the posture angle corresponding to the actual coordinates of the robot arm, the posture of the robot arm when it grabs the target object to the specified position on the assembly line, or grabs the target object from the specified position on the assembly line is obtained. This makes it possible to control the robot arm to grab the target object according to the actual coordinates and posture of the robot arm when grabbing the target object. Therefore, it is only necessary to set up a camera on the production line and use the camera to shoot the target object to quickly formulate a robot arm grabbing plan, thereby improving the efficiency of the robot arm grabbing.

[0135] Moreover, since the robot arm grasping plan can be quickly formulated based on the photos taken, the line change time is reduced, and it is also convenient to implement mixed-line production on the production line to meet the diverse production needs.

[0136] Furthermore, by correcting the first mapping relationship to obtain a second mapping relationship, the first pixel coordinate of the target object in the corresponding object photo is transformed into the first actual coordinate of the target object in the robotic arm coordinate system through the second mapping relationship. The mapping relationship between the camera coordinate system and the robotic arm coordinate system can be accurately determined, thereby improving the accuracy of the robotic arm's grasping.

[0137] In order to facilitate those skilled in the art to have a deeper understanding of the embodiments of the present application, a specific example will be used for illustration below.

[0138] This application isolates visual positioning from actual tools by setting virtual tools, replacing the cumbersome process of traditional visual positioning, and is suitable for the operation process of rapid line change and / or mixed line production on the assembly line.

[0139] refer to Figure 7 , point description:

[0140] 1. P1 is the standard storage point for products.

[0141] 2. P2 to P10 are the points selected for camera calibration product placement. Among them, the robot arm postures Rx, Ry, and Rz of the 9 points are consistent.

[0142] 3. Two new points P11 and P12 are automatically converted from point P2. X, Y, Z, Rx, and Ry are consistent with P2, but Rz is different from P2.

[0143] Figure 9 This is a flow chart of the teaching steps provided in Example 2 of this application. Figure 9 , the teaching process is as follows:

[0144] 1. From Figure 7 Grab the product at the standard product storage point P1 and place it at the teaching point P2;

[0145] 2. Record the flange center coordinates;

[0146] 3. Take photos with the camera;

[0147] 4. Frame the product's location in the image and use the framed area of ​​interest as a virtual tool.

[0148] 5. Calculate the center coordinates of the product in the camera coordinate system and the product placement angle. The calculation method is as follows:

[0149] a. Obtain the expected feature points through feature matching and record them as keypoints_obj and keypoints_scene. At the same time, obtain the product area in the image and record it as img_mat.

[0150] b. Use the above feature points keypoints_obj and keypoints_scene to calculate the transformation matrix of the two and obtain H.

[0151] c. Take the four vertices of img_mat and record them as std::vector <point2f>obj_corners.

[0152] d. After perspective transformation using H and obj_corners, the vertex positions of the template are recorded as std::vector <point2f>scene_corners.

[0153] e. Use triangulation to calculate the angle of the corresponding point using the scene_corners obtained above, as shown below:

[0154] double angelTopLeft=qtan(scene_corners[1].y-scene_corners[0].y, scene_corners[1].x-scene_corners[0].x);

[0155] double angelTopRight=qtan(scene_corners[2].y-scene_corners[3].y, scene_corners[2].x-scene_corners[3].x);

[0156] double angelBottomLeft=qtan(scene_corners[3].y-scene_corners[0].y, scene_corners[3].x-scene_corners[0].x);

[0157] double angelBottomRight=qtan(scene_corners[2].y-scene_corners[1].y,scene_corners[2].x-scene_corners[1].x);

[0158] f. Calculate the angelTopLeft, angelTopRight, angelBottomLeft, and angelBottomRight obtained above as follows:

[0159] double angleX=(angelTopLeft+angelTopRight) / 2;

[0160] double angleXY=(angelBottomLeft+angelBottomRight) / 2;

[0161] g. Use the above angleX and angleY ​​to calculate the final angle of the product, productAngle.

[0162] Double diffAngle=fabs(angleX-angleY)

[0163] diffAngle=mod(diffAngle,360)

[0164] If(diffAngle>180)

[0165] {

[0166] diffAngle -= 360;

[0167] }

[0168] Else

[0169] {

[0170] diffAngle+=360;

[0171] }

[0172] 6. Repeat steps 1, 2, 3, and 5 to grab the products to P3~P12 respectively.

[0173] 7. Use the manipulator coordinate system data and camera coordinate system data of the 9 points corresponding to P2 to P10 to calculate the rotation matrix M1 between the manipulator coordinate system and the camera coordinate system.

[0174]

[0175] in, is the representation of camera coordinates, For M1, It is the representation of the robot arm coordinates corresponding to 9 points.

[0176] 8. a. Use the camera coordinates of the three points corresponding to P2, P11, and P12, and use the principle of three points determining a circle to calculate the coordinate value of the flange center in the camera coordinate system;

[0177] b. Calculate the actual physical coordinates of the virtual tool center using the rotation matrix calculated in step 7;

[0178] c. Using the above-calculated coordinates of the virtual tool center and the actual coordinates of the flange center, calculate the offset (offsetX, offsetY) between the virtual tool and the flange center, the angle of the virtual tool, and the physical length of the virtual tool.

[0179] 9. Use the flange center coordinates of the 9 points in step 7 and the offset and angle between the tool and flange center calculated in step 8 to calculate the rotation matrix M2 between the robot coordinate system and the camera coordinate system.

[0180]

[0181] in, is the camera coordinate representation, is the rotation matrix M2, is the coordinate representation of the robot arm.

[0182] 10. Save the rotation matrix M2 of the above-mentioned robot coordinate system and camera coordinate system, the offset between the tool and the flange center (offsetX / offsetY), angle (toolAngle), virtual tool length (toolLength) and virtual tool for actual production use (production process).

[0183] Figure 10 This is a flow chart of the production steps provided in Example 2 of this application. Figure 10 , the production process is as follows:

[0184] 1. After the sensor detects that the product is in place, it starts taking pictures and obtains the product image;

[0185] 2. Get the center point and deflection angle information productCenter and productAngle of ProductImage.

[0186] 3. Correct the product position to make it consistent with the virtual tool and record the offset product0offsetX, productOffsetY, productOffsetAngle.

[0187] 4. Use the rotation matrix M2 calculated using the productCenter and teaching process above to obtain the coordinates x and y of the product center (i.e., the virtual tool center) after perspective transformation. Use the toolAngle, offsetX, and offsetY obtained through teaching to calculate the coordinates X' and Y' values ​​of the robot arm flange.

[0188] 5. Calculate the Rz value of the robot arm using the productOffsetAngle and the toolAngle and toolLength calculated from the teaching process. Specifically, obtain the pixel coordinates of the product's center point (x, y) from the photo. Then use M2 to convert the pixel coordinates of the product's center point to the robot arm's coordinates (x', y'). Take the robot arm's origin coordinates (0, 0) and calculate the angle θ between the line connecting the two points and the robot arm's x-axis. θ + the toolAngle obtained from the teaching process is the Rz value.

[0189] 6. Use the X, Y, RZ calculated in steps 5 and 6 above and the specified Rx, Ry, Z to form the complete posture of the robot arm to grab the target object to the target point.

[0190] The above-mentioned robotic arm grasping method is relatively insensitive to the installation accuracy of the tool. It does not require a complicated line change process in which each product and tool must be calibrated. The camera-tool calibration only needs to be performed once for any product model, which greatly shortens the line change time and can fully realize mixed-line production, which has a significant effect on improving production efficiency.

[0191] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0192] Example 3

[0193] Based on the same inventive concept, embodiments of the present application also provide a robotic arm grasping system for implementing the aforementioned robotic arm grasping method. The solution provided by this system is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more robotic arm grasping system embodiments provided below can be found in the above-described limitations of the robotic arm grasping method and will not be further elaborated here.

[0194] Figure 11 This is a schematic diagram of the structure of a robotic arm grasping system provided in Example 3 of this application. Figure 11 , the robotic arm grasping method provided in this embodiment specifically includes: a camera 410, a controller 420 and a robotic arm 430;

[0195] The camera 410 is used to photograph the target object and obtain an object photo corresponding to the target object;

[0196] a controller 420 configured to determine, based on a first pixel coordinate of the target object in the object photograph, a first actual coordinate of the target object in the robotic arm coordinate system, determine, based on the first actual coordinate, a second actual coordinate of the robotic arm in the robotic arm coordinate system when the robotic arm grasps the target object, determine a posture angle of the robotic arm corresponding to the second actual coordinate, and control the robotic arm to grasp the target object based on the second actual coordinate and the posture angle;

[0197] The robotic arm 430 is used to grasp the target object according to the second actual coordinates and posture angle.

[0198] In a specific implementation, a camera can be used to photograph a target object on an assembly line to obtain a photo of the object. The camera sends the photo of the object to a controller. The controller transforms the first pixel coordinate of the target object in the camera coordinate system to the first actual coordinate of the target object in the robotic arm coordinate system according to a pre-stored mapping relationship. The controller determines the second actual coordinate of the robotic arm in the robotic arm coordinate system when the robotic arm grabs the target object to a specified position on the assembly line or grabs the target object from a specified position on the assembly line according to the deviation and angle between the robotic arm and the target object in the robotic arm coordinate system. The controller determines the position of the robotic arm when the robotic arm grabs the target object to a specified position on the assembly line or grabs the target object from a specified position on the assembly line according to the second actual coordinate. The attitude angle when the robotic arm grabs the target object to the specified position on the assembly line, or grabs the target object from the specified position on the assembly line. If the robotic arm is required to grab the target object to the specified position on the assembly line, since the starting coordinates and starting attitude angle of the robotic arm are known, the path of the robotic arm can be planned and the movement of the robotic arm can be controlled based on the starting coordinates, starting attitude angle, the second actual coordinate and the attitude angle corresponding to the second actual coordinate; if the robotic arm is required to grab the target object from the specified position on the assembly line, the robotic arm can be directly controlled to grab the target object based on the second actual coordinate and the attitude angle corresponding to the second actual coordinate.

[0199] The above-mentioned robotic arm grasping system determines the first actual coordinates of the target object in the robotic arm coordinate system based on the first pixel coordinates of the target object in the corresponding object photo, determines the second actual coordinates of the robotic arm in the robotic arm coordinate system when the robotic arm grasps the target object based on the first actual coordinates, determines the attitude angle of the robotic arm corresponding to the second actual coordinates, and controls the robotic arm to grasp the target object based on the second actual coordinates and the attitude angle; the pixel coordinates of the target object in the photo can be converted into actual coordinates, and the actual coordinates when the robotic arm grasps the target object to a specified position on the assembly line or grasps the target object from a specified position on the assembly line are determined based on the actual coordinates of the target object, and the attitude angle corresponding to the actual coordinates of the robotic arm is obtained when the robotic arm grasps the target object to a specified position on the assembly line or grasps the target object from a specified position on the assembly line, so that the robotic arm can be controlled to grasp the target object based on the actual coordinates and attitude of the robotic arm grasping the target object, thereby only needing to set up a camera on the production line and photograph the target object with the camera to quickly formulate a robotic arm grasping plan, thereby improving the efficiency of robotic arm grasping.

[0200] Moreover, since the robot arm grasping plan can be quickly formulated based on the photos taken, the line change time is reduced, and it is also convenient to implement mixed-line production on the production line to meet the diverse production needs.

[0201] Example 4

[0202] Figure 12 This is a schematic diagram of the structure of a robotic arm grasping device provided in Example 4 of this application. Figure 12 The robotic arm grasping device provided in this embodiment specifically includes: a coordinate conversion module 510, a coordinate determination module 520, an angle determination module 530 and a grasping control module 540; wherein:

[0203] A coordinate conversion module 510 is configured to determine a first actual coordinate of the target object in the robot arm coordinate system according to a first pixel coordinate of the target object in the corresponding object photo;

[0204] a coordinate determination module 520 for determining, based on the first actual coordinate, a second actual coordinate of the robotic arm in the robotic arm coordinate system when the robotic arm grasps the target object;

[0205] An angle determination module 530, configured to determine a posture angle of the robotic arm corresponding to the second actual coordinate;

[0206] The grasping control module 540 is configured to control the robotic arm to grasp the target object according to the second actual coordinate and the posture angle.

[0207] In another embodiment, the robotic arm grasping device further includes:

[0208] A first mapping relationship module is used to determine a first mapping relationship between the camera coordinate system where the object photo is located and the robotic arm coordinate system;

[0209] The second mapping relationship module is used to correct the first mapping relationship to obtain a second mapping relationship between the camera coordinate system and the robotic arm coordinate system; the first pixel coordinate is transformed to the first actual coordinate through the second mapping relationship.

[0210] In another embodiment, the first mapping relationship module is further used to obtain the second pixel coordinates of the reference object in the camera coordinate system, and to obtain the third actual coordinates of the robotic arm in the robotic arm coordinate system when the robotic arm grasps the reference object; and the mapping relationship between the second pixel coordinates and the third actual coordinates is determined as the first mapping relationship between the camera coordinate system and the robotic arm coordinate system.

[0211] In another embodiment, the second mapping relationship module includes:

[0212] an actual deviation determination module, configured to determine an actual deviation between the robotic arm and the reference object in the robotic arm coordinate system;

[0213] an actual coordinate correction module, configured to correct the third actual coordinate of the robotic arm according to the actual deviation, so as to obtain a fourth actual coordinate of the reference object in the robotic arm coordinate system;

[0214] A mapping relationship determination module is used to determine the mapping relationship between the second pixel coordinate and the fourth actual coordinate as the second mapping relationship between the camera coordinate system and the robotic arm coordinate system.

[0215] In another embodiment, the actual deviation determination module includes:

[0216] a pixel coordinate module, configured to determine a third pixel coordinate of the center point of the reference object in the camera coordinate system, and to determine a fourth pixel coordinate of the center point of the robotic arm in the camera coordinate system;

[0217] an actual coordinate module, configured to determine, according to the first mapping relationship, a fifth actual coordinate corresponding to the third pixel coordinate and a sixth actual coordinate corresponding to the fourth pixel coordinate;

[0218] The deviation determining module is configured to determine the deviation between the fifth actual coordinate and the sixth actual coordinate as the actual deviation between the robotic arm and the reference object.

[0219] In another embodiment, the pixel coordinate module is also used to obtain at least three tool pixel coordinates of a tool installed at the center point of the flange device in the camera coordinate system; each of the tool pixel coordinates corresponds to a different posture angle of the tool; and the center of the circle corresponding to the at least three tool pixel coordinates is determined as the fourth pixel coordinate of the center point of the robotic arm in the camera coordinate system.

[0220] In another embodiment, the actual deviation includes a lateral deviation and a longitudinal deviation; the actual deviation determination module further includes:

[0221] a tool length determination module, configured to determine a distance between the fifth actual coordinate and the sixth actual coordinate as a tool length of a virtual tool; the virtual tool corresponding to a line connecting a center point of the reference object and a center point of the robotic arm;

[0222] A tool angle determination module is used to determine the tool angle of the virtual tool based on at least two of the lateral deviation, the longitudinal deviation and the tool length; the lateral deviation, the longitudinal deviation, the tool length and the tool angle are used to determine the second actual coordinate based on the first actual coordinate.

[0223] In another embodiment, the angle determination module 530 is further used to obtain a connecting line between the second actual coordinate and the coordinate origin of the robotic arm coordinate system; determine the angle between the connecting line and the horizontal coordinate axis of the robotic arm coordinate system; and sum the angle and the tool angle of the virtual tool to obtain the components of the posture angle.

[0224] Each module in the aforementioned robotic gripping device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a computer device's memory in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0225] Example 5

[0226] Figure 13 : It is a structural diagram of an electronic device provided in Example 5 of the present application. As shown in the figure, the electronic device includes: a processor 60, a memory 61, a display screen 62 with a touch function, an input device 63, an output device 64, and a communication device 65. The number of processors 60 in the electronic device can be one or more, and the figure takes one processor 60 as an example. The number of memories 61 in the electronic device can be one or more, and the figure takes one memory 61 as an example. The processor 60, memory 61, display screen 62, input device 63, output device 64, and communication device 65 of the electronic device can be connected via a bus or other means, and the figure takes the connection via a bus as an example. In the embodiment, the electronic device can be a computer, a mobile phone, a tablet, a projector, an interactive smart tablet, etc. In the embodiment, the electronic device is described by taking the interactive smart tablet as an example.

[0227] The memory 61, as a computer-readable storage medium, can be used to store software programs, computer executable programs, and modules, such as the program instructions / modules corresponding to the robotic arm grasping method described in any embodiment of the present application. The memory 61 may mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required for a function; the data storage area can store data created according to the use of the device, etc. In addition, the memory 61 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 61 may further include a memory remotely located relative to the processor 60, and these remote memories can be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0228] Display screen 62 is a touch-enabled display screen 62, which can be a capacitive screen, an electromagnetic screen, or an infrared screen. Generally speaking, display screen 62 is used to display data according to instructions from processor 60, and is also used to receive touch operations on display screen 62 and transmit corresponding signals to processor 60 or other devices. Optionally, when display screen 62 is an infrared screen, it also includes an infrared touch frame, which is disposed around display screen 62 and is also used to receive infrared signals and transmit them to processor 60 or other devices.

[0229] The communication device 65 is used to establish a communication connection with other devices, and can be a wired communication device and / or a wireless communication device.

[0230] The input device 63 can be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device. It can also be a camera for capturing images and a sound pickup device for capturing audio data. The output device 64 can include audio equipment such as a speaker. It should be noted that the specific composition of the input device 63 and the output device 64 can be set according to actual circumstances.

[0231] The processor 60 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 61, that is, realizes the above-mentioned robotic arm grasping method.

[0232] Specifically, in the embodiment, when the processor 60 executes one or more programs stored in the memory 61, the following operations are implemented:

[0233] Determining a first actual coordinate of the target object in the robotic arm coordinate system according to a first pixel coordinate of the target object in the corresponding object photo;

[0234] determining, based on the first actual coordinate, a second actual coordinate of the robotic arm in the robotic arm coordinate system when the robotic arm grasps the target object;

[0235] Determining the attitude angle of the robotic arm corresponding to the second actual coordinate;

[0236] The robotic arm is controlled to grasp the target object according to the second actual coordinate and the posture angle.

[0237] Based on the above embodiment, the one or more processors 60 further implement the following operations:

[0238] Determine a first mapping relationship between a camera coordinate system where the object photo is located and the robotic arm coordinate system;

[0239] The first mapping relationship is corrected to obtain a second mapping relationship between the camera coordinate system and the robotic arm coordinate system; the first pixel coordinate is transformed to the first actual coordinate through the second mapping relationship.

[0240] Acquire a second pixel coordinate of the reference object in the camera coordinate system, and acquire a third actual coordinate of the robotic arm in the robotic arm coordinate system when the robotic arm grasps the reference object;

[0241] The mapping relationship between the second pixel coordinates and the third actual coordinates is determined as the first mapping relationship between the camera coordinate system and the robotic arm coordinate system.

[0242] Based on the above embodiment, the one or more processors 60 further implement the following operations:

[0243] determining an actual deviation between the robotic arm and the reference object in the robotic arm coordinate system;

[0244] Correcting the third actual coordinate of the robotic arm according to the actual deviation to obtain a fourth actual coordinate of the reference object in the robotic arm coordinate system;

[0245] The mapping relationship between the second pixel coordinates and the fourth actual coordinates is determined as the second mapping relationship between the camera coordinate system and the robotic arm coordinate system.

[0246] Based on the above embodiment, the one or more processors 60 further implement the following operations:

[0247] Determining a third pixel coordinate of a center point of the reference object in the camera coordinate system, and determining a fourth pixel coordinate of a center point of the robotic arm in the camera coordinate system;

[0248] determining, according to the first mapping relationship, a fifth actual coordinate corresponding to the third pixel coordinate and a sixth actual coordinate corresponding to the fourth pixel coordinate;

[0249] The deviation between the fifth actual coordinate and the sixth actual coordinate is determined as the actual deviation between the robotic arm and the reference object.

[0250] Based on the above embodiment, the one or more processors 60 further implement the following operations:

[0251] Acquire at least three tool pixel coordinates of a tool mounted at the center point of the flange device in the camera coordinate system, wherein each of the tool pixel coordinates corresponds to a different posture angle of the tool;

[0252] The center of the circle corresponding to the at least three tool pixel coordinates is determined as the fourth pixel coordinate of the center point of the robotic arm in the camera coordinate system.

[0253] Based on the above embodiment, the one or more processors 60 further implement the following operations:

[0254] After determining the deviation between the fifth actual coordinate and the sixth actual coordinate as the actual deviation between the robotic arm and the reference object, the method further includes:

[0255] determining the distance between the fifth actual coordinate and the sixth actual coordinate as a tool length of a virtual tool; the virtual tool corresponding to a line connecting a center point of the reference object and a center point of the robotic arm;

[0256] The tool angle of the virtual tool is determined based on at least two of the lateral deviation, the longitudinal deviation and the tool length; the lateral deviation, the longitudinal deviation, the tool length and the tool angle are used to determine the second actual coordinate based on the first actual coordinate.

[0257] Based on the above embodiment, the one or more processors 60 further implement the following operations:

[0258] Obtaining a line between the second actual coordinate and the coordinate origin of the robotic arm coordinate system;

[0259] Determining the angle between the connecting line and the horizontal coordinate axis of the robotic arm coordinate system;

[0260] The included angle and the tool angle of the virtual tool are summed to obtain components of the posture angle.

[0261] Example 6

[0262] Embodiment 6 of the present application further provides a storage medium containing computer-executable instructions. When the computer-executable instructions are executed by a computer processor, they are used to perform a robotic arm grasping method, including:

[0263] Determining a first actual coordinate of the target object in the robotic arm coordinate system according to a first pixel coordinate of the target object in the corresponding object photo;

[0264] determining, based on the first actual coordinate, a second actual coordinate of the robotic arm in the robotic arm coordinate system when the robotic arm grasps the target object;

[0265] Determining the attitude angle of the robotic arm corresponding to the second actual coordinate;

[0266] The robotic arm is controlled to grasp the target object according to the second actual coordinate and the posture angle.

[0267] Of course, the storage medium containing computer-executable instructions provided in an embodiment of the present application, whose computer-executable instructions are not limited to the operations of the robotic arm grasping method described above, can also execute related operations in the robotic arm grasping method provided in any embodiment of the present application, and have corresponding functions and beneficial effects.

[0268] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0269] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0270] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A robotic arm grasping method, characterized in that: The method comprises: Acquire a second pixel coordinate of the reference object in the camera coordinate system, and acquire a third actual coordinate of the robotic arm in the robotic arm coordinate system when the robotic arm grasps the reference object; Determine the mapping relationship between the second pixel coordinates and the third actual coordinates as a first mapping relationship between the camera coordinate system and the robotic arm coordinate system; Determining a third pixel coordinate of a center point of the reference object in the camera coordinate system, and determining a fourth pixel coordinate of a center point of the robotic arm in the camera coordinate system; determining, according to the first mapping relationship, a fifth actual coordinate corresponding to the third pixel coordinate and a sixth actual coordinate corresponding to the fourth pixel coordinate; determining a deviation between the fifth actual coordinate and the sixth actual coordinate as an actual deviation between the robotic arm and the reference object; Correcting the third actual coordinate of the robotic arm according to the actual deviation to obtain a fourth actual coordinate of the reference object in the robotic arm coordinate system; Determine the mapping relationship between the second pixel coordinate and the fourth actual coordinate as a second mapping relationship between the camera coordinate system and the robotic arm coordinate system; determining, based on a first pixel coordinate of the target object in the corresponding object photo, a first actual coordinate of the target object in the robotic arm coordinate system; and transforming the first pixel coordinate to the first actual coordinate through the second mapping relationship; determining, based on the first actual coordinate, a second actual coordinate of the robotic arm in the robotic arm coordinate system when the robotic arm grasps the target object; Determining the attitude angle of the robotic arm corresponding to the second actual coordinate; The robotic arm is controlled to grasp the target object according to the second actual coordinate and the posture angle.

2. The method according to claim 1, characterized in that The center point of the robotic arm includes the center point of the flange device of the robotic arm; and determining the fourth pixel coordinate of the center point of the robotic arm in the camera coordinate system includes: Acquire at least three tool pixel coordinates of a tool mounted at the center point of the flange device in the camera coordinate system, wherein each of the tool pixel coordinates corresponds to a different posture angle of the tool; The center of the circle corresponding to the at least three tool pixel coordinates is determined as the fourth pixel coordinate of the center point of the robotic arm in the camera coordinate system.

3. The method according to claim 1, characterized in that The actual deviation includes lateral deviation and longitudinal deviation; After determining the deviation between the fifth actual coordinate and the sixth actual coordinate as the actual deviation between the robotic arm and the reference object, the method further includes: determining the distance between the fifth actual coordinate and the sixth actual coordinate as a tool length of a virtual tool; the virtual tool corresponding to a line connecting a center point of the reference object and a center point of the robotic arm; The tool angle of the virtual tool is determined based on at least two of the lateral deviation, the longitudinal deviation and the tool length; the lateral deviation, the longitudinal deviation, the tool length and the tool angle are used to determine the second actual coordinate based on the first actual coordinate.

4. The method according to claim 3, characterized in that Determining the attitude angle of the robotic arm corresponding to the second actual coordinate includes: Obtaining a line between the second actual coordinate and the coordinate origin of the robotic arm coordinate system; Determining the angle between the connecting line and the horizontal coordinate axis of the robotic arm coordinate system; The included angle and the tool angle of the virtual tool are summed to obtain components of the posture angle.

5. A robotic arm grasping system, characterized in that: The system includes a camera, a controller, and a robotic arm; The camera is used to photograph the target object to obtain an object photo corresponding to the target object; The controller is used to obtain the second pixel coordinates of the reference object in the camera coordinate system, and to obtain the third actual coordinates of the robotic arm in the robotic arm coordinate system when the robotic arm grasps the reference object, determine the mapping relationship between the second pixel coordinates and the third actual coordinates as the first mapping relationship between the camera coordinate system and the robotic arm coordinate system, determine the third pixel coordinates of the center point of the reference object in the camera coordinate system, and determine the fourth pixel coordinates of the center point of the robotic arm in the camera coordinate system, determine the fifth actual coordinates corresponding to the third pixel coordinates and the sixth actual coordinates corresponding to the fourth pixel coordinates according to the first mapping relationship, determine the deviation between the fifth actual coordinates and the sixth actual coordinates as the actual deviation between the robotic arm and the reference object, and determine the robotic arm according to the actual deviation. the third actual coordinate of the robotic arm being corrected to obtain the fourth actual coordinate of the reference object in the robotic arm coordinate system, determining the mapping relationship between the second pixel coordinate and the fourth actual coordinate as the second mapping relationship between the camera coordinate system and the robotic arm coordinate system, determining the first actual coordinate of the target object in the robotic arm coordinate system according to the first pixel coordinate of the target object in the object photo, transforming the first pixel coordinate to the first actual coordinate through the second mapping relationship, determining the second actual coordinate of the robotic arm in the robotic arm coordinate system when the robotic arm grasps the target object according to the first actual coordinate, determining the attitude angle of the robotic arm corresponding to the second actual coordinate, and controlling the robotic arm to grasp the target object according to the second actual coordinate and the attitude angle; The robotic arm is used to grasp the target object according to the second actual coordinate and the posture angle.

6. A robotic arm grasping device, characterized in that: The device comprises: a first mapping relationship module, configured to obtain a second pixel coordinate of the reference object in the camera coordinate system, and obtain a third actual coordinate of the robotic arm in the robotic arm coordinate system when the robotic arm grasps the reference object; and determine a mapping relationship between the second pixel coordinate and the third actual coordinate as a first mapping relationship between the camera coordinate system and the robotic arm coordinate system; a pixel coordinate module, configured to determine a third pixel coordinate of the center point of the reference object in the camera coordinate system, and to determine a fourth pixel coordinate of the center point of the robotic arm in the camera coordinate system; an actual coordinate module, configured to determine, according to the first mapping relationship, a fifth actual coordinate corresponding to the third pixel coordinate and a sixth actual coordinate corresponding to the fourth pixel coordinate; a deviation determining module, configured to determine a deviation between the fifth actual coordinate and the sixth actual coordinate as an actual deviation between the robotic arm and the reference object; an actual coordinate correction module, configured to correct the third actual coordinate of the robotic arm according to the actual deviation, so as to obtain a fourth actual coordinate of the reference object in the robotic arm coordinate system; a mapping relationship determining module, configured to determine the mapping relationship between the second pixel coordinate and the fourth actual coordinate as a second mapping relationship between the camera coordinate system and the robotic arm coordinate system; a coordinate conversion module, configured to determine a first actual coordinate of the target object in the robotic arm coordinate system based on a first pixel coordinate of the target object in the corresponding object photograph; and transforming the first pixel coordinate to the first actual coordinate through the second mapping relationship; a coordinate determination module, configured to determine, based on the first actual coordinate, a second actual coordinate of the robotic arm in the robotic arm coordinate system when the robotic arm grasps the target object; An angle determination module, configured to determine a posture angle of the robotic arm corresponding to the second actual coordinate; A grasping control module is used to control the robotic arm to grasp the target object according to the second actual coordinate and the posture angle.

7. The device according to claim 6, characterized in that The pixel coordinate module is also used to obtain at least three tool pixel coordinates of the tool installed at the center point of the flange device in the camera coordinate system; each of the tool pixel coordinates corresponds to a different posture angle of the tool; and the center of the circle corresponding to the at least three tool pixel coordinates is determined as the fourth pixel coordinate of the center point of the robotic arm in the camera coordinate system.

8. The device according to claim 6, characterized in that The device further comprises: a tool length determination module, configured to determine a distance between the fifth actual coordinate and the sixth actual coordinate as a tool length of a virtual tool; the virtual tool corresponding to a line connecting a center point of the reference object and a center point of the robotic arm; A tool angle determination module is used to determine the tool angle of the virtual tool based on at least two of the lateral deviation, the longitudinal deviation and the tool length; the lateral deviation, the longitudinal deviation, the tool length and the tool angle are used to determine the second actual coordinate based on the first actual coordinate.

9. The device according to claim 8, characterized in that The angle determination module is also used to obtain the connecting line between the second actual coordinate and the coordinate origin of the robotic arm coordinate system; determine the angle between the connecting line and the horizontal coordinate axis of the robotic arm coordinate system; and sum the angle and the tool angle of the virtual tool to obtain the components of the posture angle.

10. An electronic device, characterized in that: include: memory, one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors execute the robotic arm grasping method according to any one of claims 1 to 4.

11. A storage medium containing computer-executable instructions, characterized in that: The computer executable instructions are used to perform the robotic arm grasping method according to any one of claims 1 to 4 when executed by a computer processor.

Images