Mechanical arm control method, device and equipment and storage medium

Abstract

The invention discloses a mechanical arm control method, device and equipment and a storage medium, and relates to the technical field of industrial mechanical arms. The mechanical arm comprises at least two skeleton arms in cascade connection, the adjacent skeleton arms in the at least two skeleton arms are connected through joints, and the method comprises the steps that coordinates of a to-be-processed workpiece in a two-dimensional coordinate system are collected through a visual sensor; based on the coordinates of the to-be-processed workpiece under the two-dimensional coordinate system, the expected coordinate posture of the tail end of the mechanical arm under the three-dimensional coordinates is determined; based on the expected coordinate attitude, determining a planned motion track of the tail end of the mechanical arm moving from the initial coordinate attitude to the expected coordinate attitude; and based on the planned motion track, the tail end of the mechanical arm is controlled to move from the initial coordinate posture to the expected coordinate posture, so that the control precision of the mechanical arm is improved.

Description

Technical Field

[0001] The present application relates to the technical field of industrial robotic arms, and in particular to a robotic arm control method, device, equipment, and storage medium. Background Art

[0002] The application of robotics technology in the manufacturing industry is becoming increasingly widespread, especially in the fields of automobiles, electronic information, rail trains and aviation manufacturing.

[0003] During the robot arm trajectory operation control process, it is difficult to achieve high-precision trajectory reproduction due to reasons such as human operation and errors in the robot itself. This seriously limits the operation efficiency and quality, especially in complex operation tasks. Summary of the Invention

[0004] The embodiments of the present application provide a method, apparatus, device and storage medium for controlling a robotic arm. The technical solution provided by the present application includes the following aspects.

[0005] According to one aspect of an embodiment of the present application, a method for controlling a robotic arm is provided, wherein the robotic arm includes at least two cascaded skeletal arms, wherein adjacent skeletal arms of the at least two skeletal arms are connected by a joint, the method comprising:

[0006] The coordinates of the workpiece to be processed in the two-dimensional coordinate system are collected by the visual sensor;

[0007] Determining the desired coordinate posture of the end of the robotic arm in three-dimensional coordinates based on the coordinates of the workpiece to be processed in the two-dimensional coordinate system;

[0008] Based on the desired coordinate posture, determining a planned motion trajectory of the end of the robotic arm moving from an initial coordinate posture to the desired coordinate posture;

[0009] Based on the planned motion trajectory, the end of the robotic arm is controlled to move from the initial coordinate posture to the desired coordinate posture.

[0010] According to one aspect of an embodiment of the present application, a robotic arm control device is provided, wherein the robotic arm includes at least two skeletal arms in cascade linkage, wherein adjacent skeletal arms of the at least two skeletal arms are connected by a joint, and the device includes:

[0011] An acquisition module is used to acquire the coordinates of the workpiece to be processed in a two-dimensional coordinate system through a visual sensor;

[0012] A first determining module is configured to determine an expected coordinate posture of the end of the robotic arm in a three-dimensional coordinate system based on the coordinates of the workpiece to be processed in a two-dimensional coordinate system;

[0013] A second determining module is configured to determine, based on the desired coordinate posture, a planned motion trajectory of the end of the robotic arm moving from an initial coordinate posture to the desired coordinate posture;

[0014] A control module is used to control the end of the robotic arm to move from the initial coordinate posture to the desired coordinate posture based on the planned motion trajectory.

[0015] In one possible implementation, the control module is configured to determine, based on the planned motion trajectory, an expected coordinate posture corresponding to the t-th moment of the end of the robotic arm during the motion process, where t is a positive integer;

[0016] Determining, based on the desired coordinate pose corresponding to the t-th moment, a joint sub-pose corresponding to each joint of the robotic arm at the t-th moment, wherein the joint sub-pose corresponding to the t-th moment is characterized by a relative change relative to the joint sub-pose corresponding to the t-1-th moment;

[0017] Based on the joint sub-poses corresponding to the joints at the t-th moment, the joints are controlled so that the end of the robotic arm moves from the initial coordinate pose to the desired coordinate pose.

[0018] In one possible implementation, the control module is configured to determine the joint sub-coordinates corresponding to each joint at the t-th moment based on the joint sub-posture corresponding to each joint at the t-th moment;

[0019] Based on the joint sub-poses and the joint sub-coordinates, the joints are controlled so that the end of the robotic arm moves from the initial coordinate pose to the desired coordinate pose.

[0020] In one possible implementation, the control module is configured to determine a correspondence between a joint sub-pose of each joint of the robotic arm and the trajectory motion time based on the desired coordinate pose corresponding to the t-th moment, the initial pose corresponding to each joint, and the trajectory motion time;

[0021] According to the corresponding relationship, the joint sub-pose corresponding to each joint of the robotic arm at the t-th moment is determined.

[0022] In one possible implementation, the acquisition module is further configured to acquire the error at the t-1th moment through the visual sensor, where the error is used to represent the error between the actual coordinate posture of the end of the robotic arm and the expected coordinate posture at the t-1th moment; the control module is configured to correct the planned motion trajectory based on the error to obtain a corrected motion trajectory;

[0023] The expected coordinate posture of the end of the robotic arm corresponding to the t-th moment during the movement is determined based on the corrected motion trajectory.

[0024] In one possible implementation, the first determining module is configured to determine an expected coordinate posture of the end of the robotic arm in three-dimensional coordinates based on the coordinates of the workpiece to be processed in a two-dimensional coordinate system and a conversion relationship;

[0025] The conversion relationship is a relationship from the visual system coordinate system to the end coordinate system of the robotic arm.

[0026] According to one aspect of an embodiment of the present application, a terminal device is provided, which includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the above-mentioned robotic arm control method.

[0027] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the above-mentioned robotic arm control method.

[0028] According to one aspect of an embodiment of the present application, a computer program product is provided, which includes a computer program stored in a computer-readable storage medium. A processor reads and executes the computer program from the computer-readable storage medium to implement the above-mentioned robotic arm control method.

[0029] The technical solutions provided in the embodiments of the present application can bring the following beneficial effects:

[0030] This application determines the planned motion trajectory based on the desired coordinate posture of the end of the robotic arm, and controls the end of the robotic arm to move from the initial coordinate posture to the desired coordinate posture along the planned motion trajectory, thereby realizing automatic determination of the planned motion trajectory based on the posture data of the end of the robotic arm itself, solving the problems of manual pre-operation and low precision of the teaching method and the computer-aided design method being highly dependent on the precision and accuracy of the three-dimensional model of the robot and the workpiece, improving the precision of the robotic arm control, and thus improving the efficiency of processing the workpiece by the robotic arm. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a schematic diagram of a computer system provided by one embodiment of the present application;

[0032] Figure 2 is a flow chart of a robotic arm control method provided by one embodiment of the present application;

[0033] Figure 3This is a flowchart of the operation of a fuzzy PID controller provided by one embodiment of the present application;

[0034] Figure 4 This is a flow chart of robotic arm control provided by one embodiment of the present application;

[0035] Figure 5 This is a schematic diagram of the structure of a robotic arm control system provided by one embodiment of the present application;

[0036] Figure 6 This is a schematic diagram of a six-degree-of-freedom robotic arm provided by an embodiment of the present application;

[0037] Figure 7 is a block diagram of a robotic arm control device provided by one embodiment of the present application;

[0038] Figure 8 This is a structural block diagram of a terminal device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0040] It should be noted that the terms "first," "second," and the like in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with certain aspects of the present application as detailed in the appended claims.

[0041] First, the terms involved in this application are explained.

[0042] Robotic arm: An automated device that mimics the functions of a human arm, capable of performing various tasks such as grasping, moving, and assembling. A robotic arm typically consists of multiple links (skeletal arms), joints, an end effector (such as a gripper or spray gun), and a control system.

[0043] The skeletal arm of a robotic arm, often referred to simply as a "link" or "arm," is the primary component of the robotic arm's structure. It serves as the "skeleton" of the robotic arm, connecting the various joints to form the main structure of the robotic arm. The length, shape, and layout of the skeletal arm directly influence the range and flexibility of the robotic arm.

[0044] Joint: The movable part that connects the skeletal arms and enables relative movement between the skeletal arms.

[0045] Forward kinematics: Given the angles of each joint of the robot arm, calculating the coordinate pose of the end of the robot arm involves combining the rotation and translation transformations of each joint to form a total transformation matrix from the base of the robot arm to the end of the robot arm.

[0046] Inverse kinematics: Given the desired coordinate pose of the end of the robotic arm, calculate the joint angles required to achieve the desired coordinate pose.

[0047] Hand-eye calibration: Mount a visual sensor, such as a camera, at the end of the robotic arm and set a calibration target within the workspace, for example using the ArucoMarker method in the opencv_contrtb module. This calibration process determines the relative position and posture between the visual sensor (eye) and the end of the robotic arm (hand), thus achieving hand-eye calibration.

[0048] To further illustrate the technical solutions provided by the embodiments of the present application, this is described in detail below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of the present application provide the method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on routine or no creative labor. In steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided by the embodiments of the present application. During the actual processing process or when the control device is executed, the method can be executed in the order of the methods shown in the embodiments or drawings or in parallel.

[0049] Other features and advantages of the present application will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present application. The purposes and other advantages of the present application can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0050] Figure 1 FIG1 is a block diagram of a computer system according to an embodiment of the present application. The computer system includes a computer device 101 and a server 102 .

[0051] In one possible implementation, the computer device 101 is any electronic product that can perform human-computer interaction with an interactive object through one or more methods such as a keyboard, touchpad, touch screen, remote control, voice interaction or handwriting device, such as a PC (Personal Computer), a mobile phone, a smart phone, a PDA (Personal Digital Assistant), a wearable device, a handheld portable gaming device, a PPC (Pocket PC), a tablet computer, a laptop computer, a desktop computer, a smart car machine, a smart TV, a smart speaker, a smart watch, a car terminal, etc., but is not limited to these.

[0052] The server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network, distribution network), and big data and artificial intelligence platforms, which are not limited in this embodiment of the present application. The server 102 is directly or indirectly connected to the computer device 101 through a wired communication method or a wireless communication method, which is not limited in this application. The server 102 has a data receiving function, a data processing function and a data sending function. Of course, the server 102 can also have other functions, which are not limited in this embodiment of the present application.

[0053] Server 102 provides backend services for the client installed on computer device 101. In one possible implementation, server 102 performs primary computing tasks, while computer device 101 performs secondary computing tasks. Alternatively, server 102 performs secondary computing tasks, while computer device 101 performs primary computing tasks. Alternatively, computer device 101 and server 102 utilize a distributed computing architecture for collaborative computing.

[0054] Computer device 101 may generally refer to one of multiple computer devices. This embodiment uses computer device 101 as an example. Those skilled in the art will appreciate that the number of computer devices 101 may be greater or lesser. For example, there may be only one computer device 101, or there may be dozens, hundreds, or even more computer devices 101. This embodiment of the application does not limit the number or type of computer devices 101.

[0055] The robotic arm control method provided in the embodiment of the present application can be executed by the computer device 101, can be executed by the server 102, or can be executed interactively by the computer device 101 and the server 102, and the embodiment of the present application is not limited to this. In some embodiments, the computer device 101 can send uplink synchronization data to the server 102, and the uplink synchronization data includes the coordinates of the workpiece to be processed in the two-dimensional coordinate system collected by the visual sensor. The server 102 uses the robotic arm control method provided in the embodiment of the present application to process the coordinates of the workpiece to be processed in the two-dimensional coordinate system collected by the visual sensor sent by the computer device 101 to obtain the final planned motion trajectory, and sends downlink synchronization data to the computer device 101, and the downlink synchronization data includes the planned motion trajectory.

[0056] For example, see Figure 1, step 103, collect the coordinates of the workpiece to be processed in the two-dimensional coordinate system through the visual sensor; step 104, determine the expected coordinate posture of the end of the robot arm in the three-dimensional coordinate system based on the coordinates of the workpiece to be processed in the two-dimensional coordinate system and the conversion relationship; step 105, determine the planned motion trajectory of the end of the robot arm from the initial coordinate posture to the expected coordinate posture based on the expected coordinate posture; step 106, determine the expected coordinate posture corresponding to the end of the robot arm at the t-th moment during the movement based on the planned motion trajectory; step 107, determine the joint sub-posture corresponding to each joint of the robot arm at the t-th moment based on the expected coordinate posture corresponding to the t-th moment; step 108, determine the joint sub-coordinate corresponding to each joint at the t-th moment based on the joint sub-posture corresponding to each joint at the t-th moment; step 109, control each joint based on the joint sub-posture and joint sub-coordinate so that the end of the robot arm moves from the initial coordinate posture to the expected coordinate posture.

[0057] Those skilled in the art should understand that the above-mentioned computer device 101 and server 102 are merely examples, and other existing or future computer devices or servers, if applicable to the present application, should also be included in the scope of protection of the present application and are incorporated herein by reference.

[0058] Based on the above Figure 1 The computer system shown in the embodiment of the present application provides a shard balancing method between nodes. The method can be executed by the computer device 101, or by the server 102, or can be implemented by the interaction between the computer device 101 and the server 102. The embodiment of the present application does not limit this.

[0059] Among related technologies, robotics is increasingly being applied across various industries, such as automotive, electronics, rail vehicles, and aviation manufacturing. Compared to traditional manual design and production processes, robotics offers advantages such as greater efficiency, precision, safety, and intelligence. With the development of robotics, industrial robots are increasingly being used in manufacturing processes such as grinding and welding. As multi-degree-of-freedom robotic devices based on electronics, mechanics, and control technologies, robotic arms offer advantages such as high flexibility and low labor costs, contributing to green and intelligent manufacturing. In recent years, machine vision technology has garnered widespread attention in trajectory planning. Based on visual guidance, visual sensors capture contour images, and image processing is used to obtain the robot's target trajectory data, reducing reliance on three-dimensional models of the robot and workpiece, as well as their relative positions.

[0060] For typical robot integration applications, the main methods for controlling robot arm trajectory operations include teach-in programming and offline programming. With teach-in programming, the robot's trajectory points are typically set on-site in the corresponding robot teach pendant. Due to factors such as human operation and the robot's own errors, high-precision trajectory reproduction is difficult to achieve, severely limiting operational efficiency and quality, especially in complex tasks. Offline programming, on the other hand, does not require online programming of the robot's trajectory in a similar manner to teach-in programming, so it has significant advantages in complex operating environments. Currently, offline programming methods can be divided into computer-aided design-based and vision-based programming methods. Computer-aided design-based offline programming methods rely heavily on the precision and accuracy of the three-dimensional models of the robot and workpiece. In actual work, when the workpiece position deviates from its original position or there are interference factors, the operational accuracy will be affected.

[0061] The embodiment of the present application is described by taking the method executed by a computer device as an example. The computer device can be a computer device or a server. Figure 2 As shown, the robotic arm control method provided in the embodiment of the present application may include at least one of the following steps 201 to 204, wherein the robotic arm includes at least two cascaded skeletal arms, and adjacent skeletal arms in the at least two skeletal arms are connected by joints.

[0062] In step 201, the coordinates of the workpiece to be processed in a two-dimensional coordinate system are collected by a visual sensor.

[0063] A visual sensor is a device used to obtain image information of the workpiece to be processed and convert it into electrical signals or digital signals. It can quickly and accurately capture image data of the workpiece to be processed or the working scene of the robotic arm, providing a basis for subsequent analysis, measurement and control.

[0064] Vision sensors can be installed at different locations on the robotic arm. They can be mounted directly on the robotic arm's end effector, allowing the sensor to move with the end effector. This makes them suitable for applications requiring close, dynamic observation of workpiece details. For example, during assembly operations, the end of the robotic arm, carrying a vision sensor, can be brought close to the workpiece for precise assembly. Alternatively, the vision sensor can be fixed at a specific location within the workspace, observing the entire work area from a relatively fixed perspective. This makes it suitable for locating and inspecting workpieces within a large area. This application does not restrict the mounting location of the vision sensor on the robotic arm.

[0065] The workpiece to be processed is a workpiece that needs to be further processed, assembled, tested or other operations during the manufacturing or processing process.

[0066] For example, the body panels in the automobile manufacturing process are the workpieces to be processed, and the formed panels are welded to the body frame by a robotic arm; or, the video of a food production line shows that different types of food are sorted into corresponding packaging areas by a robotic arm.

[0067] Optionally, the coordinates of the workpiece to be processed in a two-dimensional coordinate system are collected by a visual sensor, including: collecting image information of the workpiece to be processed by a visual sensor; determining the workpiece features of the workpiece to be processed based on the image information and an edge detection algorithm; and determining the coordinates of the workpiece to be processed in a two-dimensional coordinate system based on the workpiece features.

[0068] Optionally, the workpiece to be processed includes at least two workpiece areas, and collecting the coordinates of the workpiece to be processed in a two-dimensional coordinate system by using a visual sensor includes: collecting the coordinates of the at least two workpiece areas in the two-dimensional coordinate system respectively by using a visual sensor.

[0069] Through refined extraction, the robot arm can have an accurate understanding of the operating trajectory of each workpiece area, thereby improving the accuracy and reliability of the robot arm's operation. Multi-area batch processing allows the robot arm to flexibly adapt to different work tasks and workpiece shapes, improving the adaptability and flexibility of the system; through batch processing, the operation process can be optimized, making the robot arm more efficient when performing tasks and reducing unnecessary movement and waiting time.

[0070] Optionally, the method for determining the at least two workpiece areas includes at least one of the following: determining the at least two workpiece areas based on the geometric shape of the workpiece to be processed; determining the at least two workpiece areas based on the requirements of the work task; and determining the at least two workpiece areas based on the field of view of the visual sensor.

[0071] Optionally, the method further includes: arranging and storing at least two workpiece regions in a matrix form.

[0072] Exemplarily, the workpiece to be processed is divided into at least two workpiece areas named as: Vtew1, Vtew2, Vtew3, ..., VtewX.

[0073] The coordinates of the workpiece to be processed are collected in a 2D coordinate system and hand-eye calibration is performed on the robotic arm before it performs its task. The coordinate data points on the surface of the workpiece to be processed are located in the vision system coordinate system of the vision sensor. Hand-eye calibration converts this data into the robotic arm coordinate system. A calibration code is affixed to the end of the robotic arm. After calibration, the vision system uses the vision sensor to collect the coordinates of the workpiece to be processed in a 2D coordinate system.

[0074] In step 202 , based on the coordinates of the workpiece to be processed in the two-dimensional coordinate system, the desired coordinate posture of the end of the robot arm in the three-dimensional coordinate system is determined.

[0075] The desired coordinate pose is the position (coordinates) and orientation (pose) of the end of the robotic arm in three-dimensional space after completing processing of the workpiece. It includes the three position coordinates (X, Y, Z) and three pose angles of the end of the robotic arm relative to the robotic arm's reference coordinate system, where the robotic arm's reference coordinate system can be the robotic arm's base coordinate system.

[0076] The three attitude angles can be understood as roll angle, pitch angle and yaw angle, or Euler angle.

[0077] In step 203 , based on the desired coordinate posture, a planned motion trajectory of the end of the robotic arm from the initial coordinate posture to the desired coordinate posture is determined.

[0078] The initial coordinate posture is the position (coordinate) and direction (posture) of the end of the robot arm in the three-dimensional space before processing the workpiece to be processed.

[0079] A planned motion trajectory is a collection of consecutive points that the end of the robotic arm passes through as it moves along a planned path in three-dimensional space. This trajectory reflects the spatial path of the end of the robotic arm as it handles a workpiece, such as grasping, moving, placing, or machining it.

[0080] In step 204 , based on the planned motion trajectory, the end of the robotic arm is controlled to move from the initial coordinate posture to the desired coordinate posture.

[0081] The end of the robotic arm moves from the initial coordinate posture to the desired coordinate posture, that is, according to the planned motion trajectory, the movement of each joint of the robotic arm is controlled to ensure that the end of the robotic arm moves along the planned motion trajectory.

[0082] This application determines the planned motion trajectory based on the desired coordinate posture of the end of the robotic arm, and controls the end of the robotic arm to move from the initial coordinate posture to the desired coordinate posture along the planned motion trajectory, thereby realizing automatic determination of the planned motion trajectory based on the posture data of the end of the robotic arm itself, solving the problems of manual pre-operation and low precision of the teaching method and the computer-aided design method being highly dependent on the precision and accuracy of the three-dimensional model of the robot and the workpiece, improving the precision of the robotic arm control, and thus improving the efficiency of processing the workpiece by the robotic arm.

[0083] The conversion of the two-dimensional coordinates of the workpiece to be processed into three-dimensional coordinates.

[0084] In some embodiments, based on the coordinates of the workpiece to be processed in the two-dimensional coordinate system, the expected coordinate posture of the end of the robotic arm in the three-dimensional coordinate system is determined, including: based on the coordinates of the workpiece to be processed in the two-dimensional coordinate system and the conversion relationship, the expected coordinate posture of the end of the robotic arm in the three-dimensional coordinate system is determined; wherein the conversion relationship is the relationship from the visual system coordinate system to the end coordinate system of the robotic arm.

[0085] Since the visual sensor collects a two-dimensional image, the pixel position (x, y) of the workpiece to be processed in the image is determined by the two-dimensional image. The pixel position is the coordinate in the two-dimensional coordinate system. However, the robot arm is in the three-dimensional real world, so it is necessary to convert the coordinates (x, y) in the two-dimensional coordinate system into the desired coordinate posture (P) in the three-dimensional coordinate system. x , P y , P z ).

[0086] The establishment of the relationship from the vision system coordinate system to the end coordinate system of the robot arm requires the integration and calibration of the two systems so that the vision system and the end coordinate system of the robot arm can work together.

[0087] Optionally, the visual sensor includes at least one of a binocular sensor and a monocular sensor.

[0088] For example, a binocular sensor simulates human binocular vision, using two cameras to observe the same object from different angles and calculate parallax to determine the desired three-dimensional coordinate pose of the robotic arm's end point. A monocular vision system, on the other hand, uses a single camera to capture images and, based on known workpiece dimensions or other sensor data, infers the desired three-dimensional coordinate pose of the workpiece.

[0089] Optionally, the transformation relationship includes a transformation matrix, and based on the coordinates of the workpiece to be processed in the two-dimensional coordinate system and the transformation relationship, the expected coordinate posture of the end of the robotic arm in the three-dimensional coordinate system is determined, including: based on the coordinates of the workpiece to be processed in the two-dimensional coordinate system and the transformation matrix, the expected coordinate posture of the end of the robotic arm in the three-dimensional coordinate system is determined.

[0090] The transformation matrix expression represents the conversion relationship between 2D image coordinates (i.e., the coordinates of the workpiece to be processed in the 2D coordinate system, as captured by the robot's vision sensor) and point coordinates in 3D space (i.e., the desired coordinate pose of the end of the robotic arm in 3D coordinates). The transformation matrix combines the intrinsic and extrinsic parameters of the camera to achieve the conversion from 3D world coordinates to 2D image coordinates.

[0091]

[0092] in, is the coordinate vector of the workpiece to be processed in the two-dimensional coordinate system, and x and y represent the horizontal and vertical positions of the coordinate point on the workpiece to be processed on the image plane, respectively. Contains the focal length f of the camera in pixels, which converts the 3D coordinates into normalized image coordinates. The matrix consists of a rotation matrix R and a translation vector T, which is used to describe the posture of the two-dimensional coordinate system relative to the robotic arm coordinate system.

[0093] The pinhole imaging model is the fundamental principle of the vision system. It describes how light from an object is projected onto the imaging plane through the center of the camera's optical axis. In this model, every point on the object is projected onto the image plane through the camera's optical center, forming an image. Using this model, along with the camera's intrinsic parameters (such as focal length and principal point coordinates) and extrinsic parameters (such as the camera's coordinate pose relative to the manipulator base), a transformation relationship is established between the vision system and the manipulator's end coordinate system.

[0094] The matrix R is a 3x3 matrix used to describe the rotation of the two-dimensional coordinate system relative to the robot coordinate system. The translation vector T is a 3x1 vector used to describe the position of the origin of the two-dimensional coordinate system in the robot coordinate system.

[0095] is the coordinate vector in three-dimensional coordinates, where P x 、P y 、P z Respectively represent the x, y, and z coordinates of the point in the robot coordinate system. c It is the depth information in the two-dimensional coordinate system, that is, the distance from the point to the camera; Used to normalize 3D coordinates so they can be expressed in 2D image coordinates.

[0096] Optionally, based on the coordinates of the workpiece to be processed in the two-dimensional coordinate system and the transformation matrix, the expected coordinate posture of the end of the robotic arm in the three-dimensional coordinate system is determined, including: based on the coordinates of the workpiece to be processed in the two-dimensional coordinate system, the intrinsic parameters and extrinsic parameters of the camera, the expected coordinate posture of the end of the robotic arm in the three-dimensional coordinate system is determined.

[0097] In summary, the process of converting the two-dimensional coordinates of the workpiece to be processed into three-dimensional coordinates obtains the desired coordinate posture that the end of the robot arm needs to reach.

[0098] The joint sub-pose of each joint is determined according to the coordinate pose of the end of the robotic arm.

[0099] Since the coordinates of the workpiece in the 2D coordinate system represent at least two coordinate points, the above steps result in at least two converted 3D coordinate points. Next, a continuous, smooth motion trajectory is generated between these discrete 3D coordinate points. This trajectory determines the joint sub-pose of each joint of the robot arm at time t, over the entire time period from the starting point to the end point.

[0100] In some embodiments, determining a joint sub-pose corresponding to each joint of the robotic arm at time t based on a desired coordinate pose corresponding to time t includes: determining a correspondence between the joint sub-poses of each joint of the robotic arm and the trajectory motion time based on the desired coordinate pose corresponding to time t, the initial pose corresponding to each joint, and the trajectory motion time; and determining the joint sub-pose corresponding to each joint of the robotic arm at time t based on the correspondence. The trajectory motion time is the time required for the end of the robotic arm to move from a starting position to an ending position of the planned motion trajectory.

[0101] Optionally, the method for determining the trajectory movement time includes at least one of the following: determining the trajectory movement time according to the area of ​​the workpiece to be processed; determining the trajectory movement time according to the length of the trajectory; determining the trajectory movement time according to the historical operation data of the robot arm.

[0102] For example, the client is used as a carrier and Matrix Laboratory (MATLAB) or MATLAB function Simulink is used as software to build a planned motion trajectory control system. First, a controller is established in the planned motion trajectory control system, and the operating trajectory coordinate information of the controller is used as the planned motion trajectory coordinate; then the planned motion trajectory is programmed through a trajectory planning algorithm program to simulate the planned motion trajectory.

[0103] The correspondence between the joint sub-poses of each joint of the robotic arm and the trajectory motion time can be understood as determining the relationship between the sub-poses of each joint of the robotic arm and the trajectory motion time based on the trajectory planning algorithm. Through the correspondence, the sub-pose of each joint corresponding to any moment in the trajectory motion time can be determined.

[0104] In some embodiments, based on a planned motion trajectory, the end of the robotic arm is controlled to move from an initial coordinate posture to an expected coordinate posture, including: based on the planned motion trajectory, determining the expected coordinate posture corresponding to the end of the robotic arm at the t-th moment during the motion process, where t is a positive integer; according to the expected coordinate posture corresponding to the t-th moment, determining the joint sub-posture corresponding to each joint of the robotic arm at the t-th moment, the joint sub-posture corresponding to the t-th moment is characterized by a relative change relative to the joint sub-posture corresponding to the t-1-th moment; based on the joint sub-posture corresponding to each joint at the t-th moment, controlling each joint so that the end of the robotic arm moves from the initial coordinate posture to the expected coordinate posture.

[0105] The planned motion trajectory of the continuous end of the robotic arm planned in the previous step (that is, the coordinate posture of the end of the robotic arm at each moment) is used to determine the angles (that is, joint sub-postures) that each joint of the robotic arm needs to rotate in order for the end of the robotic arm to reach the corresponding coordinate posture at each moment through the inverse kinematics model.

[0106] Optionally, a cubic polynomial is used to describe the corresponding relationship between the sub-posture θ(t) of each joint of the robotic arm and the time t.

[0107] θ(0)=a0=θ0 Formula (1)

[0108]

[0109] θ'(0)=a1=0 Formula (3)

[0110]

[0111] Formula (1) to Formula (4) are the trajectory planning algorithms. In order to achieve smooth motion of the robot arm, the trajectory function θ(t) of each joint of the robot arm needs to satisfy at least four constraints, where θ(0) = θ0 represents the initial angle θ0 of each joint of the robot arm at time t = 0; θ(t f )=θ f , indicating that at time t=t f When the end angle of the robot arm reaches angle θ f ; θ'(0)=0, indicating that at the beginning of the movement, the speed of each joint of the robot arm is zero, that is, starting from rest; θ'(t f )=0, which means that at the end of the movement, the speed of each joint of the robot arm is zero, that is, it stops smoothly.

[0112] The parameters a0, a1, a2, and a3 are the coefficients of the cubic polynomial, which are determined by satisfying the above constraints. This results in the corresponding relationship between the sub-pose θ(t) of each joint of the manipulator and time t.

[0113] Based on the joint sub-pose of each joint, the sub-coordinate of each joint is determined.

[0114] In some embodiments, based on the joint sub-posture corresponding to each joint at the tth moment, each joint is controlled so that the end of the robotic arm moves from the initial coordinate posture to the desired coordinate posture, including: based on the joint sub-posture corresponding to each joint at the tth moment, determining the joint sub-coordinate corresponding to each joint at the tth moment; based on the joint sub-posture and the joint sub-coordinate, controlling each joint so that the end of the robotic arm moves from the initial coordinate posture to the desired coordinate posture.

[0115] See formula (5), which is a parameter representation (Denavit-Hartenberg, DH) of the coordinate system transformation relationship between two adjacent joints in a robotic arm (for example, the i-th joint and the i-1-th joint).

[0116] By using formula (5), the joints of the manipulator are connected to each other to represent the transformation matrix between the end coordinate system and the base coordinate system of the manipulator in the manipulator system.

[0117]

[0118] Among them, θ i Indicates the connecting rod angle; d i Represents the link offset, which represents the translation distance along the Z axis of the i-th joint coordinate system; α i represents the torsion angle of the connecting rod; R represents the rotation transformation, T represents the translation transformation; c is the abbreviation of cos, s is the abbreviation of sin, R x (α i-1 ) represents a rotation around the X axis α i-1 The rotation matrix of the angle; T x (a i-1 ) represents a translation along the X axis i-1 The translation matrix of the distance; R z (θ i ) represents the rotation θ around the Z axis i The rotation matrix of the angle; T z (d i ) represents a translation d along the Z axis i The translation matrix of the distance.

[0119] Based on the above formula (5), the transformation from one joint coordinate system to the next joint coordinate system is described, including rotation and translation. By multiplying the transformation matrices of all joints, the total transformation matrix from the base coordinate system to the coordinate system of the end of the robot arm can be obtained, see formula (6).

[0120]

[0121] Formula (6) describes the total transformation matrix of the manipulator from the base coordinate system to the end of the manipulator: It is done by dividing the local transformation matrix between each joint Multiplied together. Represents the multiplication operation, that is, the product of all local transformation matrices from the first joint to the nth joint; n represents the number of joints of the manipulator; n x , n y ,,n z Represents the unit vector component of the Z axis of the end coordinate system of the manipulator in the base coordinate system; x , o y , o z Represents the unit vector component of the Y axis of the end coordinate system of the manipulator in the base coordinate system; a x , a y , a z Represents the unit vector component of the X-axis of the end effector coordinate system of the manipulator in the base coordinate system; p x , p y , p z Represents the position coordinates of the origin of the robot's end effector coordinate system in the base coordinate system.

[0122] Correct the movement trajectory.

[0123] In some embodiments, the method further includes: collecting the error at the t-1th moment through a visual sensor, the error being used to characterize the error between the actual coordinate posture of the end of the robotic arm and the expected coordinate posture at the t-1th moment; determining the expected coordinate posture corresponding to the tth moment of the end of the robotic arm during the movement based on the planned motion trajectory, including: correcting the planned motion trajectory based on the error to obtain a corrected motion trajectory; determining the expected coordinate posture corresponding to the tth moment of the end of the robotic arm during the movement based on the corrected motion trajectory.

[0124] Error is a measure of the difference between actual and expected results, representing the deviation between the actual coordinate pose of the end-of-arm and the expected coordinate pose. Error can arise from a variety of factors, including manufacturing errors in the manipulator, measurement errors in sensors, calculation errors in the control system, and environmental factors.

[0125] In a vision sensor, error detection is achieved by comparing the actual coordinate pose of the end-point of the robotic arm at time t-1 with a preset desired coordinate pose. The vision sensor captures the current coordinate pose of the end-point of the robotic arm and feeds this information back to the control system. Based on this feedback, the control system calculates the error—the difference between the actual value and the target value.

[0126]

[0127] Formula (7) is used to calculate the position error e(t) of the end of the robot arm at the tth moment. This error is the difference between the actual position (x, y, z) and the expected position (P x , P y , P z ) is the Euclidean distance between them.

[0128] In a robotic arm control system, the error e(t) is fed back into control, such as a proportional-integral-derivative (PID) controller. The control system then modifies the planned motion trajectory based on the error, generating a corrected trajectory to minimize the error and keep the end of the robotic arm as close to the desired trajectory as possible. Real-time monitoring and adjustments can improve the accuracy and efficiency of robotic arm operations.

[0129] A modified trajectory is a new trajectory created by adjusting the originally planned trajectory based on the difference between the actual and expected motion during the robot's movement. This correction ensures that the end-of-arm maneuvers can more accurately follow the intended target, thereby improving operational accuracy and efficiency.

[0130]

[0131] Formula (8) is the PID control formula, which is used to calculate the control input u(t) based on the error e(t). The control input is adjusted through the three parts of proportional (P), integral (I) and differential (D) to reduce the error. p e(t) is the proportional part, which is directly proportional to the error. The larger the error, the larger the control input. It is the integral part, which is proportional to the accumulation of error and can eliminate steady-state error; It is the differential part, which is proportional to the rate of change of the error. It predicts the future trend of the error and improves the response speed and stability of the system.

[0132] Optionally, the planned motion trajectory is corrected based on the error to obtain a corrected motion trajectory, including: calculating the control input u(t) using a PID formula according to the error, and updating the motion trajectory of the robot arm according to the control input u(t) to obtain a corrected motion trajectory.

[0133] For example, assume that the robotic arm needs to move from point A to point B in a straight line. During the movement, due to external interference, the actual path of the robotic arm deviates from the planned motion trajectory. The visual sensor detects the actual coordinate posture of the end of the robotic arm, and the error between it and the expected coordinate posture is calculated. The PID control formula is used to calculate the control input that needs to be adjusted at each time step. If the error is large, the proportional part will increase the control input, the integral part will adjust according to the accumulation of the error, and the differential part will predict and adjust according to the rate of change of the error; by updating the motion trajectory of the robotic arm in real time and adjusting it according to the PID control input, it is ensured that the end of the robotic arm can accurately move along the corrected motion trajectory and finally reach the position of the workpiece to be processed.

[0134] Optionally, the number of visual sensors is at least two, and the at least two visual sensors include at least one horizontal visual sensor placed horizontally and at least one longitudinal visual sensor placed vertically; collecting the error at the t-1th moment through the visual sensors includes: collecting the lateral error at the t-1th moment through at least one horizontal visual sensor, and collecting the longitudinal error at the t-1th moment through at least one longitudinal visual sensor; correcting the planned motion trajectory based on the errors to obtain a corrected motion trajectory, including: performing a horizontal correction on the planned motion trajectory based on the lateral error, performing a longitudinal correction on the planned motion trajectory based on the longitudinal error, and obtaining a corrected motion trajectory based on the lateral correction and the longitudinal correction.

[0135] Horizontal and vertical vision sensors capture deviation information from two different directions as the robot arm's end-of-arm handles the workpiece. Through two independent corrections, the robot arm can more precisely adjust its end-of-arm position, ensuring that the workpiece is placed in the precise, predetermined location. This dual correction mechanism reduces cumulative error and improves positioning accuracy.

[0136] For example, if a robotic arm needs to screw a screw into a hole in a steel plate on an assembly line, the lateral vision sensor detects the horizontal deviation of the screw relative to the hole. Based on this, the end of the robotic arm performs a lateral correction in the X-axis direction, adjusting its position to align with the center of the hole. The longitudinal vision sensor detects the vertical deviation of the screw relative to the hole. Based on this, the end of the robotic arm performs a longitudinal correction in the Y-axis direction, further adjusting its position. Ultimately, the lateral correction in the X-axis direction and the longitudinal correction in the Y-axis direction form a corrected motion trajectory.

[0137] For example, see Figure 3The following figure shows the operation flow chart of a fuzzy PID controller. A fuzzy PID controller is an advanced control strategy that combines fuzzy logic with PID control. It leverages the flexibility of fuzzy logic and the stability of PID control to improve control system performance. The fuzzy PID controller includes fuzzification 301, fuzzy inference 302, and defuzzification 303.

[0138] Fuzzification 301 is the process of converting precise input values ​​into fuzzy sets. In control systems, the input values ​​are typically the error e(t) and the rate of change of the error e'(t). The precise values ​​are mapped into fuzzy sets, such as "negative large," "negative small," "zero," "positive small," "positive large," etc. The fuzzification process involves defining input membership functions 304, which describe the degree to which the input value belongs to each fuzzy set. For example, the error e(t) and the rate of change of the error e'(t) can be represented by triangular or trapezoidal membership functions, respectively. The value of the input membership function 304 ranges between 0 and 1, indicating the degree to which the input value belongs to a fuzzy set.

[0139] Fuzzy reasoning 302 is the process of reasoning about fuzzified inputs based on rule base 305. Rule base 305 contains a series of if-then rules that define the relationship between input fuzzy sets and output fuzzy sets. For example, a fuzzy rule might be "If the error is negative and large and the rate of change of the error is negative and small, then the control output is proportional gain large."

[0140] Defuzzification 303 is the process of converting the fuzzy output set obtained by fuzzy inference 302 into an accurate control output value. Since the fuzzy output set is a fuzzy value, it needs to be converted into a specific value based on the output membership function 306 in order to be used for actual control.

[0141] Through the above steps, a fuzzy PID controller is designed in Matlab / Simulink to achieve the goal of motion trajectory control. The fuzzy PID controller can dynamically adjust the PID parameters based on fuzzy logic reasoning about the error and the rate of error change, thereby improving the robustness and adaptability of the control system.

[0142] For example, see Figure 4In the robotic arm control flow chart shown, "Machine vision system connection successful" 401 and "Robotic arm system connection successful" 402 are the starting points of the process, ensuring that the two key systems are ready to work together; "Real-time robotic arm system operation trajectory diagram" 403 indicates that the robotic arm system is operating according to the preset or real-time calculated planned motion trajectory; "Planned motion trajectory information" 404 provides the expected trajectory that the end of the robotic arm needs to follow; "Real-time operation trajectory information" 405 is the trajectory actually followed by the end of the robotic arm during execution, which has an error with the "Planned motion trajectory information" 404 and needs to be adjusted by the control system; "Whether the control system is" 406 indicates that the control system is monitoring and comparing the "Planned motion trajectory information" 404 with the real-time trajectory, and making corrections based on the error.

[0143] For example, see Figure 5 The schematic diagram of the structure of the robotic arm control system is shown. The robotic arm control system includes a trajectory control system 501, a robotic arm system 502 and a visual system 503.

[0144] The trajectory control system 501 includes a planned motion trajectory 504 and a controller 505. The planned motion trajectory 504 is the desired trajectory that the robotic arm control system expects the end of the robotic arm to follow; the controller 505 is the core of the robotic arm control system. It receives the planned motion trajectory and calculates the control signal based on the error 508 between the actual coordinate posture 512 transmitted by the vision system 503 and the desired coordinate posture.

[0145] The robotic arm system 502 includes an operation program input 506, a robotic arm operation 507, and an actual coordinate posture 512. The operation program input 506 is used to input the control signal generated by the controller 505 into the robotic arm system 502 to guide the movement of the robotic arm; the robotic arm operation 507 is used to perform corresponding actions according to the received control signal, such as rotation, translation, etc., to realize the planned motion trajectory 504; the actual coordinate posture 512 is the coordinate posture of the trajectory actually traversed by the end of the robotic arm in space.

[0146] The visual system 503 includes an actual position 509, an image processing module 510, and a camera 511. The actual position 509 is the actual three-dimensional coordinate of the end of the robotic arm output by the image processing module, which is used to compare with the planned motion trajectory 504 and calculate the error 508; the image processing module 510 is responsible for extracting the position information of the end of the robotic arm from the image captured by the camera; the camera 511 is used to capture the actual position of the end of the robotic arm, and through the image processing module 510, the image data is converted into the three-dimensional coordinates of the end of the robotic arm.

[0147] For example, see Figure 6The figure shows a schematic diagram of a six-degree-of-freedom robotic arm. Each joint and skeleton arm in the figure is defined by parameters, including the joint angle (θ), the skeleton arm length (a), and the skeleton arm offset (d). The figure shows six joints (revolute joints), each represented by a circle 601, and the skeleton arms between the joints are represented by rectangles 602. Each joint can rotate around its axis, thereby driving the skeleton arm to move. Each joint has its own local coordinate system, represented by three coordinate axes X, Y, and Z. The base coordinate system X w , Y w , Z w The starting reference coordinate system 603 of the robot arm is represented by (X1-X6, Y1-Y6, Z1-Z6); a1 is the distance along the X-axis of the previous coordinate system, describing the horizontal or axial offset of the adjacent skeleton arms (e.g., a1 is the distance from {X1} to {X2} along the X1 axis), and a2 and a3 are similar to a1; d1 is the distance along the Z-axis of the previous coordinate system, describing the vertical or axial offset of the adjacent skeleton arms (e.g., d1 is the distance from {X1} to {X2} along the X1 axis). w} to {X1} along Z w The same applies to d5, d6 and d1.

[0148] This application is relatively inexpensive to use, has good compatibility, and can be used flexibly. It can plan and control the operation trajectory for workpieces of various sizes and types, meeting the needs of users with different types of operations. This application uses visual image acquisition to plan the motion trajectory and the actual operation trajectory of the robot arm, and can perform real-time trajectory operation control, thereby reducing operation errors and adapting to trajectory planning for workpieces of different structures, types, and materials. It solves the problems of the robot arm processing trajectory, the manual pre-operation and low precision of the teaching method, and the computer-aided design method that relies heavily on the precision and accuracy of the three-dimensional model of the robot and the workpiece.

[0149] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0150] Please refer to Figure 7 , which shows a block diagram of a robotic arm control provided by one embodiment of the present application. The robotic arm includes at least two skeletal arms in cascade linkage, wherein adjacent skeletal arms of the at least two skeletal arms are connected by joints. The device includes:

[0151] The acquisition module 701 is used to acquire the coordinates of the workpiece to be processed in a two-dimensional coordinate system through a visual sensor;

[0152] A first determining module 702 is configured to determine an expected coordinate posture of the end of the robotic arm in a three-dimensional coordinate system based on the coordinates of the workpiece to be processed in a two-dimensional coordinate system;

[0153] The second determining module 703 is used to determine the planned motion trajectory of the end of the robotic arm from the initial coordinate posture to the desired coordinate posture based on the desired coordinate posture;

[0154] The control module 704 is used to control the end of the robotic arm to move from the initial coordinate posture to the desired coordinate posture based on the planned motion trajectory.

[0155] In one possible implementation, the control module 704 is configured to determine, based on the planned motion trajectory, an expected coordinate posture corresponding to the t-th moment of the end of the robotic arm during the motion process, where t is a positive integer;

[0156] According to the expected coordinate posture corresponding to the t-th moment, the joint sub-posture corresponding to each joint of the manipulator at the t-th moment is determined. The joint sub-posture corresponding to the t-th moment is represented by the relative change relative to the joint sub-posture corresponding to the t-1-th moment.

[0157] Based on the joint sub-pose corresponding to each joint at the tth moment, each joint is controlled so that the end of the robotic arm moves from the initial coordinate pose to the desired coordinate pose.

[0158] In one possible implementation, the control module 704 is configured to determine the joint sub-coordinates corresponding to each joint at time t based on the joint sub-pose corresponding to each joint at time t;

[0159] Based on the joint sub-poses and joint sub-coordinates, each joint is controlled so that the end of the robotic arm moves from the initial coordinate pose to the desired coordinate pose.

[0160] In one possible implementation, the control module 704 is configured to determine a correspondence between a joint sub-pose of each joint of the robotic arm and a trajectory motion time based on the desired coordinate pose corresponding to the t-th moment, the initial pose corresponding to each joint, and the trajectory motion time;

[0161] According to the corresponding relationship, the joint sub-pose corresponding to each joint of the robotic arm at the tth moment is determined.

[0162] In one possible implementation, the acquisition module 701 is further configured to acquire an error at the t-1th moment through a visual sensor, where the error is used to represent the error between the actual coordinate posture of the end of the robotic arm and the desired coordinate posture at the t-1th moment; the control module 704 is configured to correct the planned motion trajectory based on the error to obtain a corrected motion trajectory;

[0163] The desired coordinate posture of the end of the robotic arm corresponding to the t-th moment during the motion process is determined based on the corrected motion trajectory.

[0164] In one possible implementation, the first determining module 702 is configured to determine an expected coordinate posture of the end of the robotic arm in a three-dimensional coordinate system based on the coordinates of the workpiece to be processed in a two-dimensional coordinate system and a transformation relationship;

[0165] The conversion relationship is the conversion relationship from the visual system coordinate system to the end coordinate system of the robot arm.

[0166] It should be noted that the apparatus provided in the above embodiments, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0167] An embodiment of the present application also provides a computer device, which includes: a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the robotic arm control method provided by the above-mentioned method embodiments.

[0168] For example, Figure 8 is a block diagram of a computer device 1000 provided by an exemplary embodiment of the present application. The computer device 1000 may be Figure 1 The computer device shown is used to implement the robotic arm control method provided in the above embodiment.

[0169] The computer device 1000 includes a central processing unit (CPU) 1001, a system memory 1004 including a random access memory (RAM) 1002 and a read-only memory (ROM) 1003, and a system bus 1005 connecting the system memory 1004 and the CPU 1001. The computer device 1000 also includes a basic input / output system (I / O system) 1006 for facilitating information transmission between various components within the computer device, and a mass storage device 1007 for storing an operating system 1013, application programs 1014, and other program modules 1015.

[0170] The basic input / output system 1006 includes a display 1008 for displaying information and an input device 1009, such as a mouse or keyboard, for user input. Both the display 1008 and the input device 1009 are connected to the central processing unit 1001 via an input / output controller 1010 connected to the system bus 1005. The basic input / output system 1006 may also include an input / output controller 1010 for receiving and processing input from a variety of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1010 also provides output to a display screen, printer, or other types of output devices.

[0171] The mass storage device 1007 is connected to the central processing unit 1001 via a mass storage controller (not shown) connected to the system bus 1005. The mass storage device 1007 and its associated computer-readable storage medium provide non-volatile storage for the computer device 1000. In other words, the mass storage device 1007 may include a computer-readable storage medium (not shown) such as a hard disk or a Compact Disc Read-Only Memory (CD-ROM) drive.

[0172] Without loss of generality, the computer-readable storage medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable storage instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electronically Erasable Programmable Read-Only Memory (EEPROM), flash memory or other solid-state storage devices, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, tape cassettes, magnetic tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage media are not limited to the above. The above-mentioned system memory 1004 and mass storage device 1007 can be collectively referred to as memory.

[0173] The memory stores one or more programs, and the one or more programs are configured to be executed by one or more central processing units 1001. The one or more programs contain instructions for implementing the above-mentioned method embodiments. The central processing unit 1001 executes the one or more programs to implement the robotic arm control method provided by the above-mentioned method embodiments.

[0174] According to various embodiments of the present application, the computer device 1000 may also be connected to a remote computer device on a network such as the Internet for operation. That is, the computer device 1000 may be connected to the network 1012 via the network interface unit 1011 connected to the system bus 1005, or the network interface unit 1011 may be used to connect to other types of networks or remote computer device systems (not shown).

[0175] The memory also includes one or more programs, which are stored in the memory and include steps executed by a computer device in the robotic arm control method provided in an embodiment of the present application.

[0176] A computer-readable storage medium is also provided in an embodiment of the present application, which stores at least one instruction, at least one program, code set or instruction set. When the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor of a computer device, the robotic arm control method provided by the above-mentioned method embodiments is implemented.

[0177] The present application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the robotic arm control method provided by each of the above method embodiments.

[0178] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned computer-readable storage medium may be a read-only memory, a disk or an optical disk, etc.

[0179] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent switches, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for controlling a robotic arm, characterized in that: The robotic arm includes at least two cascaded skeletal arms, adjacent skeletal arms of the at least two skeletal arms are connected by joints, and the method includes: The coordinates of the workpiece to be processed in the two-dimensional coordinate system are collected by the visual sensor; Determining the desired coordinate posture of the end of the robotic arm in three-dimensional coordinates based on the coordinates of the workpiece to be processed in the two-dimensional coordinate system; Based on the desired coordinate posture, determining a planned motion trajectory of the end of the robotic arm moving from an initial coordinate posture to the desired coordinate posture; Based on the planned motion trajectory, the end of the robotic arm is controlled to move from the initial coordinate posture to the desired coordinate posture.

2. The method according to claim 1, characterized in that The step of controlling the end of the robotic arm to move from the initial coordinate posture to the desired coordinate posture based on the planned motion trajectory includes: Based on the planned motion trajectory, determining the expected coordinate posture of the end of the robotic arm corresponding to the t-th moment during the motion process, where t is a positive integer; Determining, based on the desired coordinate pose corresponding to the t-th moment, a joint sub-pose corresponding to each joint of the robotic arm at the t-th moment, wherein the joint sub-pose corresponding to the t-th moment is characterized by a relative change relative to the joint sub-pose corresponding to the t-1-th moment; Based on the joint sub-poses corresponding to the joints at the t-th moment, the joints are controlled so that the end of the robotic arm moves from the initial coordinate pose to the desired coordinate pose.

3. The method according to claim 2, characterized in that The controlling of each joint based on the joint sub-pose corresponding to each joint at the t-th moment so that the end of the robotic arm moves from the initial coordinate pose to the desired coordinate pose includes: Determine the joint sub-coordinates corresponding to each joint at the t-th moment based on the joint sub-poses corresponding to each joint at the t-th moment; Based on the joint sub-poses and the joint sub-coordinates, the joints are controlled so that the end of the robotic arm moves from the initial coordinate pose to the desired coordinate pose.

4. The method according to claim 2 or 3, characterized in that The determining, based on the expected coordinate posture corresponding to the t-th moment, the joint sub-postures corresponding to the joints of the robotic arm at the t-th moment includes: Determine a correspondence between a joint sub-pose of each joint of the robotic arm and the trajectory motion time according to the expected coordinate pose corresponding to the t-th moment, the initial pose corresponding to each joint, and the trajectory motion time; According to the corresponding relationship, the joint sub-pose corresponding to each joint of the robotic arm at the t-th moment is determined.

5. The method according to claim 2, characterized in that The method further comprises: Collecting the error at the t-1th moment by the visual sensor, wherein the error is used to represent the error between the actual coordinate posture of the end of the robotic arm and the expected coordinate posture at the t-1th moment; The step of determining the desired coordinate posture corresponding to the t-th moment of the end of the robotic arm during the movement based on the planned motion trajectory includes: Correcting the planned motion trajectory based on the error to obtain a corrected motion trajectory; The expected coordinate posture of the end of the robotic arm corresponding to the t-th moment during the movement is determined based on the corrected motion trajectory.

6. The method according to any one of claims 1 to 5, characterized in that: The step of determining the desired coordinate posture of the end of the robotic arm in three-dimensional coordinates based on the coordinates of the workpiece to be processed in the two-dimensional coordinate system includes: Determining the desired coordinate posture of the end of the robotic arm in three-dimensional coordinates based on the coordinates of the workpiece to be processed in the two-dimensional coordinate system and the conversion relationship; The conversion relationship is a relationship from the visual system coordinate system to the end coordinate system of the robotic arm.

7. A robotic arm control device, characterized in that: The robotic arm comprises at least two skeletal arms in cascade linkage, wherein adjacent skeletal arms of the at least two skeletal arms are connected by joints, and the device comprises: An acquisition module is used to acquire the coordinates of the workpiece to be processed in a two-dimensional coordinate system through a visual sensor; A first determining module is configured to determine an expected coordinate posture of the end of the robotic arm in a three-dimensional coordinate system based on the coordinates of the workpiece to be processed in a two-dimensional coordinate system; A second determining module is configured to determine, based on the desired coordinate posture, a planned motion trajectory of the end of the robotic arm moving from an initial coordinate posture to the desired coordinate posture; A control module is used to control the end of the robotic arm to move from the initial coordinate posture to the desired coordinate posture based on the planned motion trajectory.

8. A terminal device, characterized in that: The terminal device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the robotic arm control method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the robotic arm control method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product includes a computer program, which is stored in a computer-readable storage medium. A processor reads and executes the computer program from the computer-readable storage medium to implement the robot arm control method according to any one of claims 1 to 6.

Images