Mechanical arm control method and device, electronic equipment and computer program product

Abstract

The embodiment of the invention is suitable for the technical field of mechanical arm control, and provides a mechanical arm control method and device, electronic equipment and a computer program product. The method comprises the steps that first pixel coordinates of a first target object clamped by a mechanical arm are obtained; acquiring a second pixel coordinate of a second target object at the target moving position of the mechanical arm; according to the first pixel coordinates and the second pixel coordinates, movement information of the mechanical arm on the plane is determined; controlling the mechanical arm to move on the plane according to the moving information; if the position relation between the first target object and the second target object meets the assembling condition, the mechanical arm is controlled to move in the preset direction according to the first preset force, so that assembling of the first target object and the second target object is completed; the assembly condition is that the projection of the first target object on the plane of the second target object is overlapped with the second target object. According to the method provided by the embodiment, the accuracy of executing the assembly operation by using the mechanical arm can be improved.

Description

Technical Field

[0001] The embodiments of the present application belong to the field of robotic arm control technology, and in particular, relate to a robotic arm control method, device, electronic device, and computer program product. Background Art

[0002] In the industrial field, there are many repetitive tasks, such as shaft-hole alignment operations. With the continuous advancement of robotics technology, a large number of robotic arms have been introduced into industrial scenarios to replace human labor in performing such repetitive tasks. Currently, most robotic arm control solutions are designed for structured application scenarios. In structured application scenarios, the initial shape of the target object remains fixed, and its positional relationship with the target position remains unchanged. Therefore, in existing robotic arm control solutions, the robotic arm often operates according to a fixed trajectory to complete the given task. For example, the robotic arm will grasp the target object from point A according to a preset posture and then move it to point B to complete the task.

[0003] However, in real factory environments, workpieces often roll and shift, causing random changes in the initial shape of the target object and the positional relationship between the target object and the target position. Therefore, in practical applications, existing robotic arm control solutions may fail to accurately complete the designated task, resulting in low operational accuracy. Summary of the Invention

[0004] In view of this, embodiments of the present application provide a robotic arm control method, device, electronic device, and computer program product to improve the accuracy of robotic arm operation.

[0005] A first aspect of an embodiment of the present application provides a robotic arm control method, comprising:

[0006] Obtaining a first pixel coordinate of a first target object gripped by the robotic arm;

[0007] Acquire second pixel coordinates of a second target object at the target movement position of the robotic arm;

[0008] determining movement information of the robotic arm on a plane according to the first pixel coordinates and the second pixel coordinates;

[0009] controlling the robotic arm to move on the plane according to the movement information;

[0010] If the positional relationship between the first target and the second target meets the assembly condition, the robotic arm is controlled to move in a preset direction according to a first preset force to complete the assembly of the first target and the second target; the assembly condition is that the projection of the first target on the plane where the second target is located overlaps with the second target.

[0011] In a possible implementation manner of the first aspect, the movement information includes a first movement speed of the robotic arm in an X direction of the plane and a second movement speed of the robotic arm in a Y direction of the plane;

[0012] The determining, based on the first pixel coordinates and the second pixel coordinates, movement information of the robotic arm on the plane includes:

[0013] calculating the first moving speed according to a difference between the abscissa in the first pixel coordinates and the abscissa in the second pixel coordinates;

[0014] The second moving speed is calculated according to the difference between the vertical coordinate in the first pixel coordinates and the vertical coordinate in the second pixel coordinates.

[0015] In a possible implementation of the first aspect, controlling the robotic arm to move on the plane according to the movement information includes:

[0016] The robotic arm is controlled to move on the plane according to the first moving speed, the second moving speed, and a preset time.

[0017] In a possible implementation of the first aspect, controlling the robotic arm to move in a preset direction according to the first preset force includes:

[0018] Controlling the robotic arm to move in a preset direction according to a first preset force, and obtaining a first feedback force in the preset direction;

[0019] If the first feedback force is within a preset reference force range, controlling the robotic arm to stop moving in the preset direction;

[0020] controlling the robot arm to move in the X direction and / or the Y direction of the second plane according to the second preset force, and obtaining a second feedback force; wherein the direction of the second feedback force is opposite to the direction of the second preset force;

[0021] If the second feedback force is greater than or equal to a first threshold, the robotic arm is controlled to release the first target.

[0022] In a possible implementation of the first aspect, if the positional relationship between the first target object and the second target object satisfies an assembly condition, controlling the robotic arm to move in a preset direction according to a first preset force to complete assembly of the first target object and the second target object includes:

[0023] Acquire a second image including the first object and the second object;

[0024] Performing target detection on the second image to obtain a third coordinate of the first target object in the second image and a fourth coordinate of the second target object;

[0025] If the difference between the horizontal coordinate in the third coordinate and the horizontal coordinate in the fourth coordinate is less than a second threshold, and the difference between the vertical coordinate in the third coordinate and the vertical coordinate in the fourth coordinate is less than a third threshold, the robotic arm is controlled to move in a preset direction according to a first preset force to complete the assembly of the first target object and the second target object.

[0026] In a possible implementation manner of the first aspect, before acquiring the first pixel coordinates of the first target object clamped by the robotic arm, the method includes:

[0027] Performing target detection on the third image containing the first target object to obtain initial pose parameters corresponding to the first target object;

[0028] generating motion parameters according to the initial pose parameters and target pose parameters corresponding to the first target object;

[0029] The robotic arm is controlled to grasp the first target object according to the motion parameters.

[0030] In a possible implementation of the first aspect, when the first target object is a workpiece with a hole, the second target object is a workpiece with a pin; when the first target object is a workpiece with a pin, the second target object is a workpiece with a hole.

[0031] A second aspect of an embodiment of the present application provides a robotic arm control device, comprising:

[0032] A first coordinate acquisition module, used to acquire a first pixel coordinate of a first target object clamped by the robotic arm;

[0033] A second coordinate acquisition module, configured to acquire second pixel coordinates of a second target object at a target moving position of the robotic arm;

[0034] a movement information determining module, configured to determine movement information of the robotic arm on a plane based on the first pixel coordinates and the second pixel coordinates;

[0035] a moving module, configured to control the movement of the robotic arm on the plane according to the movement information;

[0036] An assembly module is configured to control the movement of the robotic arm in a preset direction according to a first preset force to complete assembly of the first target and the second target if the positional relationship between the first target and the second target satisfies an assembly condition; the assembly condition being that the projection of the first target on the plane where the second target is located overlaps with the second target.

[0037] A third aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the robotic arm control method as described in the first aspect above is implemented.

[0038] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the robotic arm control method as described in the first aspect above.

[0039] A fifth aspect of the embodiments of the present application provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the robotic arm control method described in the first aspect.

[0040] Compared with the prior art, the embodiments of the present application have the following advantages:

[0041] In an embodiment of the present application, the electronic device can determine the movement information of the robotic arm on the plane based on the first pixel coordinates of the first target object clamped by the robotic arm and the second pixel coordinates of the second target object at the target movement position of the robotic arm, and control the movement of the robotic arm on the plane based on the movement information so that the positional relationship between the first target object and the second target object meets the assembly condition. Through the method provided by this embodiment, since the electronic device can generate the movement information of the robotic arm on the plane based on the first pixel coordinates and the second pixel coordinates, that is, the movement trajectory of the robotic arm on the plane is not fixed, but is generated by the electronic device based on the relationship between the first pixel coordinates and the second pixel coordinates. When the position of the first target object changes, the first pixel coordinates will also change; similarly, when the position of the second target object changes, the second pixel coordinates will also change. The electronic device can plan the movement trajectory based on the changed first pixel coordinates and / or second pixel coordinates, so that the robotic arm can control the movement of the first target object based on the changed position of the second target object, thereby ensuring the precise assembly of the first target object and the second target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] To more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0043] Figure 1 is a schematic diagram of a robotic arm control method provided in an embodiment of the present application;

[0044] Figure 2 is a schematic diagram of an assembly process provided in an embodiment of the present application;

[0045] Figure 3 is a schematic diagram of another robotic arm control method provided in an embodiment of the present application;

[0046] Figure 4 is a schematic diagram of a robotic arm control device provided in an embodiment of the present application;

[0047] Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present application with unnecessary details.

[0049] In industrial production, especially in the equipment manufacturing and assembly industries, numerous scenarios involving the automated docking, assembly, and maintenance of equipment parts often occur, such as in factory workshops, warehouse logistics centers, or outdoor assembly sites. These scenarios often require workers to perform numerous repetitive tasks, such as shaft-hole alignment. With the increasing demand for both unmanned and automated operations in industrial production, a large number of robotic arms have been introduced to industrial scenarios to replace human labor in performing repetitive tasks.

[0050] Among them, the scenario for performing the shaft hole alignment operation mainly includes two characteristics. Feature 1. Unstructured: There are many types of workpieces that need to perform shaft hole alignment operations, and the sizes, shapes and materials of the workpieces are different; in addition, the workspace is complex, so the movement route of the robot arm may be more complicated; further, since the workpiece may roll, shift, etc., the placement of the two workpieces that need to be assembled is often random. It can be seen that the execution scenario for performing the shaft hole alignment operation is not a fixed structured scenario, but an ever-changing unstructured scenario. Feature 2. High-precision requirements: Shaft hole alignment requires extremely high precision, and the error is usually in millimeters. Due to the above two characteristics of the execution scenario of the shaft hole alignment operation, when the control scheme in the existing technology that controls the robot arm to complete the task according to the preset trajectory is applied to the shaft hole alignment operation, it may happen that the robot arm cannot accurately complete the given task, resulting in low accuracy of the robot arm operation.

[0051] In view of this, an embodiment of the present application provides a method for controlling a robotic arm. Through the method provided by this embodiment, the electronic device can perform trajectory planning based on the correlation between the first target and the second target in the image to generate movement information. Then, the electronic device can control the robotic arm holding the first target to move above the second target based on the movement information to align the first target and the second target. When the positional relationship between the first target and the second target meets the assembly conditions, the electronic device can control the robotic arm holding the first target to move toward the second target to complete the assembly of the first target and the second target. It can be seen that the robotic arm in the embodiment of the present application is not assembled according to a preset trajectory, but is planned according to the trajectory of the image obtained by real-time shooting. Therefore, the method provided by this embodiment can improve the accuracy of the operation of the robotic arm, thereby improving the success rate of completing assembly tasks using the robotic arm.

[0052] The technical solution of this application is described below through specific embodiments.

[0053] Reference Figure 1 , showing a schematic diagram of a method for controlling a robotic arm provided in an embodiment of the present application. The method can be applied to electronic devices assembled inside a robot, such as a microcontroller unit (MCU), a microprocessor (MPU), a digital signal processor (DSP), a field programmable gate array (FPGA), a system on chip (SoC), etc.; the method can also be applied to electronic devices that are arranged outside the robot but have a communication link with the robotic arm for controlling the robotic arm, such as computers, tablet computers, mobile terminals, servers, etc. The above-mentioned method for controlling the robotic arm can specifically include the following steps:

[0054] S101 , obtaining first pixel coordinates of a first target object gripped by a robotic arm.

[0055] In this embodiment, when a user requires the robotic arm to perform an assembly operation, the user can issue an assembly instruction to the electronic device. In response to the user's assembly instruction, the electronic device can control the robotic arm on the robot to grasp the first target object corresponding to the assembly instruction. After controlling the robotic arm to grasp the first target object, the electronic device can control the robotic arm to continue holding the first target object with a second preset force.

[0056] Among them, the robotic arm can grasp the first target object through the actuator at the end of the robotic arm. Grasping can be used to indicate that the robotic arm moves to the vicinity of the first target object, and performs actions such as clamping or adsorbing the first target object quickly and accurately to achieve initial contact and control of the target object. Grasping can be used to emphasize the instantaneous action of the robotic arm to obtain the first target object, for example, the process of the clamping claw at the end of the robotic arm quickly closing to clamp a workpiece. Holding can be used to indicate that after the robotic arm has grasped the first target object, it continues to maintain the clamping or adsorption state of the target object with a certain force to ensure that the target object will not fall or move during subsequent operations. Holding can be used to emphasize the continuous state of the robotic arm maintaining control over the target object, for example, after the clamping claw grasps the workpiece, it clamps the workpiece tightly with an appropriate force so that the workpiece can be stably clamped in the clamping claw when the robotic arm moves or performs other operations. When the assembly instruction initiated by the user is an axis-hole alignment instruction, the first target object may be a workpiece with a hole, in which case the second target object may be a workpiece with a pin; the first target object may also be a workpiece with a hole, in which case the second target object may be a workpiece with a pin.

[0057] When the robotic arm is holding the first target, the electronic device can capture a first image containing the first target and the second target through the camera on the robot. Specifically, the electronic device can capture the first image through a camera installed on the wrist of the robotic arm, and / or capture the first image through a camera installed on the head of the robot. The electronic device can perform target detection on the first target in the first image and obtain the first pixel coordinates of the first target. The first pixel coordinates may be the coordinates of the target assembly position of the first target in the pixel coordinate system of the first image. Specifically, when the first target is a workpiece with a hole and the second target is a workpiece with a pin, the first pixel coordinates may be the coordinates of the hole on the first target in the pixel coordinate system of the first image. When the first target is a workpiece with a pin and the second target is a workpiece with a hole, the first pixel coordinates may be the coordinates of the pin on the first target in the pixel coordinate system of the first image.

[0058] In one possible implementation, before controlling the robotic arm to grasp the first target, the electronic device may first capture a third image containing the first target through the camera on the robotic arm wrist and / or the camera on the robot head. After acquiring the third image, the electronic device may perform target detection on the first target in the third image to obtain initial posture parameters corresponding to the first target. Specifically, the initial posture parameters may include the current initial position parameters and initial posture parameters of the first target. The initial position parameters can be used to indicate the current specific position of the first target in space. The initial posture parameters can be used to indicate the current orientation of the first target in space, which can be expressed by Euler angles (yaw angle, pitch angle, roll angle). After acquiring the current initial posture parameters of the first target, the electronic device may generate action parameters based on the target posture parameters of the first target pre-set by the R&D personnel. Then, the electronic device may control the robotic arm to grasp the first target based on the action parameters, so that the robotic arm holds the first target according to the target posture.

[0059] S102: Acquire second pixel coordinates of a second target object at the target moving position of the robotic arm.

[0060] In this embodiment, after the electronic device acquires a first image containing a first target and a second target, it may perform target detection on the second target in the first image and obtain first pixel coordinates of the second target at the target movement position of the robotic arm. The second pixel coordinates may be the coordinates of the target assembly position of the second target in the pixel coordinate system of the first image. The target assembly position may be the location used to connect the two targets during assembly.

[0061] Specifically, when the first object is a workpiece with a hole and the second object is a workpiece with a pin, the second pixel coordinates may be the coordinates of the pin on the second object in the pixel coordinate system of the first image. When the first object is a workpiece with a pin and the second object is a workpiece with a hole, the second pixel coordinates may be the coordinates of the hole on the second object in the pixel coordinate system of the first image.

[0062] S103 : Determine movement information of the robotic arm on the plane according to the first pixel coordinate and the second pixel coordinate.

[0063] In this embodiment, after obtaining the first pixel coordinates and the second pixel coordinates, the electronic device can perform trajectory planning based on the first pixel coordinates and the second pixel coordinates to generate movement information for controlling the robotic arm on a plane. Specifically, the movement information generated by the electronic device can be movement information of the robotic arm on a first plane. For example, when the second target object is a workpiece placed on an operating table, the first plane can be a plane parallel to the operating table at a certain height above the operating table.

[0064] In one possible implementation, after obtaining the first pixel coordinates and the second pixel coordinates, the electronic device may generate movement information of the robotic arm on the plane based on the position difference between the first pixel coordinates and the second pixel coordinates. Specifically, the movement information generated by the electronic device may include a first movement speed of the robotic arm in the X direction of the plane and a second movement speed of the robotic arm in the Y direction of the plane.

[0065] In one possible implementation, when the first target is a workpiece with a hole and there is only one hole on the first target, and the second target is a workpiece with a pin and there is only one pin on the second target, the electronic device can obtain a first pixel coordinate and a second pixel coordinate. At this time, the electronic device can calculate the first moving speed based on the difference between the horizontal coordinate in the first pixel coordinate and the horizontal coordinate in the second pixel coordinate. The electronic device can also calculate the second moving speed based on the difference between the vertical coordinate in the first pixel coordinate and the vertical coordinate in the second pixel coordinate. After obtaining the first moving speed and the second moving speed, the electronic device can control the robot arm to move on the plane according to the first moving speed, the second moving speed and the preset time length, so that the first target reaches above the second target, and the hole on the first target is aligned with the pin on the second target.

[0066] Specifically, the formula for the electronic device to calculate the first moving speed and the second moving speed may be as follows.

[0067] v x =pd1·(x1-x2)

[0068] v y =pd2·(y1-y2)

[0069] Where p is a conversion coefficient used to express the proportional relationship between visual data and movement speed. d1 is the first correction coefficient corresponding to the X direction. d2 is the second correction coefficient corresponding to the Y direction. x It can represent the first moving speed corresponding to the X direction. y may represent the second moving speed in the Y direction. x1 may represent the horizontal coordinate in the first pixel coordinate. x2 may represent the horizontal coordinate in the second pixel coordinate. y1 may represent the vertical coordinate in the first pixel coordinate. y2 may represent the vertical coordinate in the second pixel coordinate.

[0070] In one possible implementation, after obtaining the first and second movement velocities, the electronic device may calculate a combined velocity of the first and second movement velocities based on the parallelogram rule or the triangle rule. The electronic device may then control the robotic arm to move on a plane based on a preset duration and the combined velocity.

[0071] In one possible implementation, after obtaining the first movement speed and the second movement speed, the electronic device may first control the robotic arm to move in one of the X direction or the Y direction according to the first preset duration and the movement speed corresponding to the direction, and then control the robotic arm to move in the other direction according to the second preset duration and the movement speed corresponding to the direction. For example, the electronic device may first control the robotic arm to move in the X direction of the plane according to the first preset duration and the first movement speed, and then control the robotic arm to move in the Y direction of the plane according to the second preset duration and the second movement speed.

[0072] In one possible implementation, when the first target object is a workpiece with holes and has multiple holes, and the second target object is a workpiece with pins and has multiple pins, and the number of pins is the same as the number of holes, the electronic device can obtain multiple first pixel coordinates and multiple second pixel coordinates. After obtaining the multiple first pixel coordinates and multiple second pixel coordinates, the electronic device can first generate posture parameters of the robotic arm based on the multiple first pixel coordinates and multiple second pixel coordinates, and adjust the gripping posture of the robotic arm based on the posture parameters so that the projection of the line connecting the multiple holes on the plane of the second target object is parallel to the line connecting the multiple pins.

[0073] Specifically, the electronic device may first calculate a first slope of a line connecting the plurality of holes based on the plurality of first pixel coordinates, and calculate a second slope of a line connecting the plurality of pins based on the plurality of second pixel coordinates. The electronic device may then generate the posture parameters of the robotic arm based on the difference between the first and second slopes.

[0074] After adjusting the gripping posture of the robotic arm, the electronic device can determine the pin associated with a hole on the first target object based on the preset correspondence between holes and pins, and determine the second pixel coordinates associated with the first pixel coordinate of the hole. The electronic device can then generate movement information based on the coordinate difference between the associated first pixel coordinates and the second pixel coordinates. The specific method for the electronic device to generate movement information based on a first pixel coordinate and a second pixel coordinate is described in the relevant content of this embodiment and is not further described here.

[0075] In one possible implementation, the first target object may also be a workpiece with a pin, and the second target object may be a workpiece with a hole. Readers may refer to the implementation method in the case where the first target object is a workpiece with a hole and the second target object is a workpiece with a pin, and replace the "hole" in the first target object with "pin", and replace the "pin" in the second target object with "hole" to understand the specific implementation method of this embodiment.

[0076] Through the method provided in this embodiment, since the electronic device can directly generate the movement information of the robotic arm based on the coordinates of the first target object and the second target object in the pixel coordinate system of the first image, the method provided in this embodiment enables the electronic device to generate movement information without having to perform coordinate conversion between the pixel coordinate system and the world coordinate system, and without having to further combine the internal parameters and external parameters of the camera for coordinate conversion. Therefore, the method provided in this embodiment can reduce the amount of calculation of the electronic device, thereby improving the response speed of the robotic arm.

[0077] In addition, since the movement information in this embodiment is calculated by the electronic device based on the coordinates of the target object in the pixel coordinate system, the method provided in this embodiment is used to control the robotic arm, so that the error after the first target object and the second target object are aligned can be within 1 pixel, that is, the error after the first target object and the second target object are aligned can be within 1 mm. Therefore, the method provided in this embodiment can improve the accuracy of the robotic arm in aligning the first target object and the second target object.

[0078] In one possible implementation, the electronic device may further input the first pixel coordinates and the second pixel coordinates into a trajectory planning model to generate movement information of the robotic arm on a plane using the trajectory planning model. The trajectory planning model may be obtained by training the electronic device based on a training dataset input by a user before controlling the robotic arm to complete an assembly operation. Specifically, the training dataset may include multiple first training coordinates, second training coordinates, and a desired trajectory. After obtaining the training model and training dataset, the electronic device may input the first and second training coordinates from the training dataset into the training model to generate an initial trajectory corresponding to the first and second training coordinates. The electronic device may then calculate a loss value based on the initial trajectory and the desired trajectory and determine whether the loss value is less than or equal to a loss threshold. If the electronic device determines that the loss value is greater than the loss threshold, the electronic device may update the training model based on the loss value and regenerate the initial trajectory based on the updated training model to update the training model. The electronic device may continue to perform this updating operation until the loss value is less than or equal to the loss threshold. If the electronic device determines that the loss value is greater than the loss threshold, the electronic device may stop performing the updating operation and determine the current training model as the trajectory planning model for generating movement information based on the first and second pixel coordinates. It should be noted that the trajectory planning model can be any model known to those skilled in the art for trajectory planning, such as BEVGPT, Drive-WM, Caspformer, etc. The embodiment of the present application is not intended to specifically limit the trajectory planning model.

[0079] S104: Control the robotic arm to move on the plane according to the movement information.

[0080] In this embodiment, after generating the movement information, the electronic device can control the robot arm to move on the plane according to the movement information.

[0081] S105 : If the positional relationship between the first target object and the second target object meets the assembly condition, the robot arm is controlled to move in a preset direction according to the first preset force to complete the assembly of the first target object and the second target object.

[0082] In this embodiment, after controlling the robotic arm to move on a plane based on the movement information, the electronic device can determine whether the positional relationship between the first target and the second target satisfies an assembly condition. The assembly condition can be that the projection of the first target on the plane where the second target resides overlaps with the second target. Specifically, when the first target is a workpiece with a hole and the second target is a workpiece with a pin, the assembly condition can be that the projection of the hole on the first target on the plane where the second target resides overlaps with the pin on the second target. When the first target is a workpiece with a pin and the second target is a workpiece with a hole, the assembly condition can be that the projection of the pin on the first target on the plane where the second target resides overlaps with the hole on the second target.

[0083] If the electronic device determines that the positional relationship between the first and second objects satisfies the assembly conditions, the electronic device can control the robotic arm to move in a preset direction based on a first preset force, i.e., the electronic device can control the robotic arm to approach the second object to complete the assembly of the first and second objects. The preset direction can be determined based on the assembly relationship between the first and second objects. Specifically, when the first and second objects are in an oblique assembly relationship, the preset direction can be a direction with an angle less than 90 degrees with the plane containing the second object. When the first and second objects are in a perpendicular assembly relationship, the preset direction can be a direction perpendicular to the plane containing the second object, i.e., the angle between the preset direction and the plane containing the second object can be equal to 90 degrees. The first preset force can be the externally applied force required for the robotic arm to move in the preset direction. The X and Y directions can be directions in a three-dimensional coordinate system constructed with the center of the first object as the origin.

[0084] If the electronic device determines that the positional relationship between the first target object and the second target object does not meet the assembly conditions, the electronic device can continue to acquire the first image containing the first target object and the second target object, and regenerate the movement information based on the latest acquired first image, and continue to control the movement of the robotic arm on the plane based on the new movement information until the positional relationship between the first target object and the second target object meets the assembly conditions, that is, the electronic device can return to execute S101 to S104 until the positional relationship between the first target object and the second target object meets the assembly conditions.

[0085] Through the method provided by this embodiment, because the electronic device can generate movement information of the robotic arm on a plane based on the first pixel coordinates of the first target and the second pixel coordinates of the second target collected in real time, the method provided by this embodiment enables the robotic arm to move the first target based on the actual positions of the first and second targets in the scene during an assembly task. Therefore, the method provided by this embodiment can improve the accuracy of the robotic arm's assembly operations, thereby increasing the robotic arm's success rate in completing assembly tasks.

[0086] In addition, since the electronic device will control the robotic arm to assemble the first target object and the second target object only after determining that the positional relationship between the first target object and the second target object meets the assembly conditions, the method provided in this embodiment can reduce the situation where the targets are assembled without being aligned, thereby causing damage to the targets. Therefore, the method provided in this embodiment can improve the safety of the robotic arm when performing assembly tasks.

[0087] See also Figure 2 , shows a schematic diagram of an assembly process provided by an embodiment of the present application. Figure 2 The object may include a first target 201 and a second target 204, wherein the first target 201 includes a hole 202 and the second target 204 includes a pin 203. Figure 2 As shown in (a) of FIG, the first target object 201 held by the robot arm may be a workpiece with a hole. The electronic device may control the movement of the robot arm according to the contents of S101 to S102 of this embodiment, so that the robot arm moves the first target object 201 to the top of the second target object 204 and aligns the hole 202 in the first target object 201 with the pin 203 in the second target object 204. Figure 2 As shown in (b) of the figure, after the electronic device aligns the hole 202 in the first target 201 with the pin 203 in the second target 204, since the second target 204 is located below the first target 201, the electronic device can control the robotic arm to move downward in a preset direction, thereby causing the robotic arm to move the first target 201 downward. As the robotic arm moves the first target 201 downward, the pin 203 on the second target can pass through the hole 202 in the first target 201, allowing the pin 203 on the second target 204 to be inserted into the hole 202 in the first target 201, completing the assembly of the first target 201 and the second target 204.

[0088] In one possible implementation, after controlling the movement of the robotic arm on a plane based on the movement information, the electronic device may capture a second image containing the first target and the second target through a camera on the wrist and / or head of the robotic arm. After capturing the second image, the electronic device may perform target detection on the first target and the second target in the second image to obtain a third coordinate of the first target and a fourth coordinate of the second target in the second image. The electronic device may then calculate the difference between the abscissa in the third coordinate and the abscissa in the fourth coordinate, as well as the difference between the ordinate in the third coordinate and the ordinate in the fourth coordinate.

[0089] If the electronic device determines that the difference between the horizontal coordinate in the third coordinate and the horizontal coordinate in the fourth coordinate is less than the preset second threshold, and the difference between the vertical coordinate in the third coordinate and the vertical coordinate in the fourth coordinate is less than the preset third threshold, the electronic device can determine that the current positional relationship between the first target object and the second target object meets the assembly conditions, and the electronic device can control the robotic arm to move in a preset direction to complete the assembly of the first target object and the second target object.

[0090] If the electronic device determines that the difference between the horizontal coordinate in the third coordinate and the horizontal coordinate in the fourth coordinate is greater than or equal to the preset second threshold, and / or the difference between the vertical coordinate in the third coordinate and the vertical coordinate in the fourth coordinate is greater than or equal to the preset third threshold, the electronic device can determine that the current positional relationship between the first target object and the second target object does not meet the assembly condition, and the electronic device can return to execute S101 to S102 until the positional relationship between the first target object and the second target object meets the assembly condition.

[0091] In one possible implementation, after determining that the positional relationship between the first target object and the second target object satisfies the assembly condition, the electronic device may input the first preset force into a preset torque calculation formula to calculate the torque value of the joint of the robotic arm when the robotic arm moves in a preset direction. After calculating the torque value corresponding to the first preset force, the electronic device may control the joint of the robotic arm based on the torque value to move the robotic arm in the preset direction toward the second target object.

[0092] Specifically, the torque calculation formula can be shown as follows:

[0093]

[0094] Among them, τ can represent the torque value of the robot arm joint. Z It can represent a first preset force in a preset direction. The velocity Jacobian matrix of the robotic arm can be expressed as Describes the relationship between the linear velocity of the end of the robot and the angular velocity of the robot joint. Specifically, It can be a 6xN matrix, where N can be a positive integer greater than or equal to 1. The first three lines can represent the linear speed of the end of the robot in the X direction, Y direction and preset direction respectively. The last three rows of the matrix represent the angular velocities of the robot arm's joints around the X, Y, and Z axes, respectively. The symbol [:,:3] represents the operation performed on all elements in the first three columns of the matrix. Specifically, the symbol ":" in the above formula represents all rows, and ":3" represents the first three columns.

[0095] Figure 3 FIG2 shows a specific implementation flow chart of a robot arm control method S103 provided in the second embodiment of the present application. Figure 3 , compared to Figure 1 In the embodiment, the robot arm control method provided in this embodiment includes S1051 to S1054, which are described in detail as follows:

[0096] S1051. Control the robotic arm to move in a preset direction according to a first preset force, and obtain a first feedback force in the preset direction.

[0097] In this embodiment, while controlling the movement of the robotic arm in the preset direction, the electronic device can continuously collect a first feedback force in the preset direction through a force sensor on the robotic arm. The direction of the first feedback force can be opposite to the movement direction of the robotic arm.

[0098] S1052: If the first feedback force is within a preset reference force range, control the robotic arm to stop moving in a preset direction.

[0099] In this embodiment, the electronic device can continuously determine whether the collected first feedback force is within a preset reference force range. If the electronic device determines that the first feedback force is within the reference force range, the electronic device can control the robotic arm to stop moving in the preset direction. If the electronic device determines that the first feedback force is not within the reference force range, the electronic device can continue to control the robotic arm to move in the preset direction, and continue to obtain the first feedback force in the preset direction. Among them, the reference force range can be determined by the R&D personnel based on multiple reference forces collected during the experimental stage. Specifically, during the experimental stage, the R&D personnel can use the remote control device to control the robotic arm to perform assembly actions on the first target object and the second target object. At the same time, the robotic arm can continuously collect the feedback force in this process as a reference force for fitting the reference force range.

[0100] S1053: Control the robotic arm to move in the X direction and / or Y direction of the second plane according to the second preset force, and obtain a second feedback force.

[0101] In this embodiment, after controlling the robotic arm to stop moving in the preset direction, the electronic device can control the robotic arm to move in the X direction and / or Y direction of the second plane with a second preset force, and obtain a second feedback force. The second plane can be the plane where the second target object is currently located. The second feedback force can be a force whose action direction is opposite to the action direction of the second preset force. After obtaining the second feedback force, the electronic device can determine whether the second feedback force is greater than or equal to the first threshold. If the second feedback force is less than the first threshold, the electronic device can control the robotic arm to move in the preset direction according to the third preset force, so that the first target object leaves the second target object, and re-execute the operations of S101 to S104. The action direction of the third preset force is opposite to the action direction of the first preset force.

[0102] S1054: If the second feedback force is greater than or equal to the first threshold, control the robotic arm to release the first target object.

[0103] In this embodiment, if the electronic device determines that the second feedback force is greater than or equal to the preset first threshold, the electronic device can determine that the first target and the second target are assembled successfully, and the electronic device can control the robotic arm to release the first target.

[0104] Specifically, the electronic device may first control the robot's robotic arm to move in the X-direction with a second preset force and obtain a second feedback force in the X-direction. The electronic device may determine whether the second feedback force in the X-direction is greater than or equal to a first threshold. If the electronic device determines that the second feedback force in the X-direction is less than the first threshold, the electronic device may determine that the first and second objects are not fully assembled. The electronic device may control the robotic arm to move in a preset direction based on a third preset force to separate the first object from the second object, and re-execute operations S101 to S104. If the electronic device determines that the second feedback force in the X-direction is greater than or equal to the first threshold, the electronic device may further control the robot's robotic arm to move in the Y-direction with a second preset force and obtain a second feedback force in the Y-direction. The electronic device may determine whether the second feedback force in the X-direction is greater than or equal to the first threshold. If the electronic device determines that the second feedback force in the Y-direction is less than the first threshold, the electronic device may determine that the first and second objects are not fully assembled. The electronic device may control the robotic arm to move in a preset direction based on the third preset force to separate the first object from the second object, and re-execute operations S101 to S104. If the electronic device determines that the second feedback force in the Y direction is greater than or equal to the first threshold, the electronic device can determine that the first target and the second target have been assembled, and the electronic device can control the robotic arm to release the first target.

[0105] With the method provided in this embodiment, after controlling the robotic arm to move in a preset direction, the electronic device can further determine whether the first and second objects are fully assembled based on the feedback force in the preset direction, as well as the feedback force in the X and / or Y directions, and release the first object only when the feedback force exceeds a threshold. Therefore, the method provided in this embodiment can reduce the probability of components falling or being damaged by collisions due to release before they are fully assembled, thereby improving the stability and safety of the robotic arm in performing assembly operations.

[0106] It should be noted that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0107] Reference Figure 4 , shows a schematic diagram of a robotic arm control device provided by an embodiment of the present application, which may specifically include a first coordinate acquisition module block 401, a second coordinate acquisition module 402, a movement information determination module 403, a movement module 404, and an assembly module 405, wherein:

[0108] A first coordinate acquisition module 401 is used to acquire a first pixel coordinate of a first target object held by the robotic arm;

[0109] A second coordinate acquisition module 402 is used to acquire second pixel coordinates of a second target object at the target moving position of the robotic arm;

[0110] a movement information determining module 403, configured to determine movement information of the robotic arm on a plane according to the first pixel coordinates and the second pixel coordinates;

[0111] A movement module 404, configured to control the movement of the robotic arm on the plane according to the movement information;

[0112] The assembly module 405 is used to control the movement of the robotic arm in a preset direction according to a first preset force to complete the assembly of the first target and the second target if the positional relationship between the first target and the second target meets the assembly condition; the assembly condition is that the projection of the first target on the plane where the second target is located overlaps with the second target.

[0113] The movement information determination module 403 can also be used to calculate the first movement speed based on the difference between the horizontal coordinate in the first pixel coordinate and the horizontal coordinate in the second pixel coordinate; and calculate the second movement speed based on the difference between the vertical coordinate in the first pixel coordinate and the vertical coordinate in the second pixel coordinate.

[0114] The moving module 404 may also be configured to control the movement of the robotic arm on the plane according to the first moving speed, the second moving speed, and a preset duration.

[0115] The assembly module 405 can also be used to control the robot arm to move in a preset direction according to a first preset force, and obtain a first feedback force in the preset direction; if the first feedback force is within a preset reference force range, the robot arm is controlled to stop moving in the preset direction; according to a second preset force, the robot arm is controlled to move in the X direction and / or Y direction of the second plane, and a second feedback force is obtained; the direction of action of the second feedback force is opposite to the direction of action of the second preset force; if the second feedback force is greater than or equal to a first threshold, the robot arm is controlled to release the first target object.

[0116] The assembly module 405 can also be used to obtain a second image containing the first target object and the second target object; perform target detection on the second image to obtain the third coordinate of the first target object in the second image and the fourth coordinate of the second target object; if the difference between the horizontal coordinate in the third coordinate and the horizontal coordinate in the fourth coordinate is less than the second threshold, and the difference between the vertical coordinate in the third coordinate and the vertical coordinate in the fourth coordinate is less than the third threshold, then control the robotic arm to move in a preset direction according to the first preset force to complete the assembly of the first target object and the second target object.

[0117] The first coordinate acquisition module 401 can also be used to perform target detection on a third image containing a first target object, and obtain initial posture parameters corresponding to the first target object; generate action parameters based on the initial posture parameters and the target posture parameters corresponding to the first target object; and control the robotic arm to grasp the first target object based on the action parameters.

[0118] In the first coordinate acquisition module 401 , when the first target object is a workpiece with a hole, the second target object is a workpiece with a pin; when the first target object is a workpiece with a pin, the second target object is a workpiece with a hole.

[0119] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment part.

[0120] Reference Figure 5 , shows a schematic diagram of an electronic device provided by an embodiment of the present application. Figure 5As shown, the electronic device 500 in the embodiment of the present application includes: a processor 510, a memory 520, and a computer program 521 stored in the memory 520 and executable on the processor 510. When the processor 510 executes the computer program 521, the steps in each embodiment of the above-mentioned robot arm control method are implemented, such as Figure 1 Alternatively, when the processor 510 executes the computer program 521, the functions of the modules / units in the above-mentioned device embodiments are realized, for example, Figure 4 Functions of modules 401 to 405 are shown.

[0121] Exemplarily, the computer program 521 may be divided into one or more modules / units, which are stored in the memory 520 and executed by the processor 510 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, which may be used to describe the execution process of the computer program 521 in the electronic device 500. For example, the computer program 521 may be divided into a trajectory planning module, a movement module, and an assembly module, with the specific functions of each module being as follows:

[0122] A first coordinate acquisition module, used to acquire a first pixel coordinate of a first target object clamped by the robotic arm;

[0123] A second coordinate acquisition module, configured to acquire second pixel coordinates of a second target object at a target moving position of the robotic arm;

[0124] a movement information determining module, configured to determine movement information of the robotic arm on a plane based on the first pixel coordinates and the second pixel coordinates;

[0125] a moving module, configured to control the movement of the robotic arm on the plane according to the movement information;

[0126] An assembly module is configured to control the movement of the robotic arm in a preset direction according to a first preset force to complete assembly of the first target and the second target if the positional relationship between the first target and the second target satisfies an assembly condition; the assembly condition being that the projection of the first target on the plane where the second target is located overlaps with the second target.

[0127] The electronic device 500 may be a microcontroller unit (MCU), a microprocessor unit (MPU), a digital signal processor (DSP), a field programmable gate array (FPGA), a system on chip (SoC), or other devices. The electronic device 500 may also be a computer, a tablet computer, a mobile terminal, a server, or other devices. The electronic device 500 may include, but is not limited to, a processor 510 and a memory 520. Those skilled in the art will understand that Figure 5 It is only an example of the electronic device 500 and does not constitute a limitation of the electronic device 500. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device 500 may also include input and output devices, network access devices, buses, etc.

[0128] The processor 510 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0129] The memory 520 may be an internal storage unit of the electronic device 500, such as a hard disk or memory of the electronic device 500. The memory 520 may also be an external storage device of the electronic device 500, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 500. Furthermore, the memory 520 may include both an internal storage unit of the electronic device 500 and an external storage device. The memory 520 is used to store the computer program 521 and other programs and data required by the electronic device 500. The memory 520 may also be used to temporarily store data that has been output or is about to be output.

[0130] An embodiment of the present application also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the robotic arm control method described in the aforementioned embodiments is implemented.

[0131] An embodiment of the present application further discloses a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the robot arm control method as described in the above embodiments is implemented.

[0132] An embodiment of the present application further discloses a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the robot arm control method described in each of the aforementioned embodiments.

[0133] The above embodiments are intended only to illustrate the technical solutions of the present application and are not intended to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they may still modify the technical solutions described in the above embodiments or replace some of the technical features therein with equivalents; and such modifications or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application and should be included within the scope of protection of the present application.

Claims

1. A method for controlling a robotic arm, characterized in that: include: Obtaining a first pixel coordinate of a first target object gripped by the robotic arm; Acquire a second pixel coordinate of a second target object at the target moving position of the robotic arm; determining movement information of the robotic arm on a plane according to the first pixel coordinates and the second pixel coordinates; controlling the robotic arm to move on the plane according to the movement information; If the positional relationship between the first target object and the second target object meets the assembly condition, controlling the robotic arm to move in a preset direction according to a first preset force to complete the assembly of the first target object and the second target object; The assembly condition is that a projection of the first target on the plane where the second target is located overlaps with the second target.

2. The method according to claim 1, characterized in that The movement information includes a first movement speed of the robotic arm in the X direction of the plane and a second movement speed of the robotic arm in the Y direction of the plane; The determining, based on the first pixel coordinates and the second pixel coordinates, movement information of the robotic arm on the plane includes: calculating the first moving speed according to a difference between the abscissa in the first pixel coordinates and the abscissa in the second pixel coordinates; The second moving speed is calculated according to the difference between the vertical coordinate in the first pixel coordinates and the vertical coordinate in the second pixel coordinates.

3. The method according to claim 2, characterized in that The controlling the robotic arm to move on the plane according to the movement information includes: The robotic arm is controlled to move on the plane according to the first moving speed, the second moving speed, and a preset time.

4. The method according to claim 1, wherein The controlling the robotic arm to move in a preset direction according to the first preset force includes: Controlling the robotic arm to move in a preset direction according to a first preset force, and obtaining a first feedback force in the preset direction; If the first feedback force is within a preset reference force range, controlling the robotic arm to stop moving in the preset direction; controlling the robot arm to move in the X direction and / or the Y direction of the second plane according to the second preset force, and obtaining a second feedback force; wherein the direction of the second feedback force is opposite to the direction of the second preset force; If the second feedback force is greater than or equal to a first threshold, the robotic arm is controlled to release the first target.

5. The method according to claim 1, wherein If the positional relationship between the first target object and the second target object satisfies an assembly condition, controlling the robotic arm to move in a preset direction according to a first preset force to complete the assembly of the first target object and the second target object includes: Acquire a second image including the first object and the second object; Performing target detection on the second image to obtain a third coordinate of the first target object in the second image and a fourth coordinate of the second target object; If the difference between the horizontal coordinate in the third coordinate and the horizontal coordinate in the fourth coordinate is less than a second threshold, and the difference between the vertical coordinate in the third coordinate and the vertical coordinate in the fourth coordinate is less than a third threshold, the robotic arm is controlled to move in a preset direction according to a first preset force to complete the assembly of the first target object and the second target object.

6. The method according to any one of claims 1 to 5, characterized in that Before obtaining the first pixel coordinates of the first target object held by the robotic arm, the method includes: Performing target detection on the third image containing the first target object to obtain initial pose parameters corresponding to the first target object; generating motion parameters according to the initial pose parameters and target pose parameters corresponding to the first target object; The robotic arm is controlled to grasp the first target object according to the motion parameters.

7. The method according to any one of claims 1 to 5, characterized in that When the first target object is a workpiece with a hole, the second target object is a workpiece with a pin; when the first target object is a workpiece with a pin, the second target object is a workpiece with a hole.

8. A robotic arm control device, characterized in that: include: A first coordinate acquisition module, used to acquire a first pixel coordinate of a first target object clamped by the robotic arm; A second coordinate acquisition module, configured to acquire second pixel coordinates of a second target object at a target moving position of the robotic arm; a movement information determining module, configured to determine movement information of the robotic arm on a plane based on the first pixel coordinates and the second pixel coordinates; a moving module, configured to control the movement of the robotic arm on the plane according to the movement information; an assembly module, configured to control the robotic arm to move in a preset direction according to a first preset force to complete assembly of the first target and the second target if the positional relationship between the first target and the second target meets an assembly condition; The assembly condition is that a projection of the first target on the plane where the second target is located overlaps with the second target.

9. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the robotic arm control method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The invention comprises a computer program, which enables the robot arm control method according to any one of claims 1 to 7 to be executed when the computer program is executed.

Images