Control method and system of mechanical arm, electronic equipment, medium and program product

Abstract

The invention discloses a control method and system of a mechanical arm, electronic equipment, a medium and a program product. The control method comprises the steps that the target pose of an end effector under world coordinates and forward kinematics information of the mechanical arm are obtained; according to the first joint angle of the mechanical arm, the forward kinematics information and the first current pose of the end effector, the second current pose of the end effector under the world coordinates is determined; determining a loss function, wherein the loss function comprises a first loss function corresponding to the difference between the target pose and the second current pose; and based on the loss function, the first joint angle is updated through a gradient descent algorithm to control the mechanical arm to move. The mechanical arm is controlled to move without collecting training data in advance, and the calculation cost is reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of robotics, and in particular to a control method, system, electronic device, medium, and program product for a robotic arm. Background Art

[0002] There are many different control algorithms for robotic motion paths, suitable for different types of robotic systems and environments. Generally speaking, these control algorithms can be divided into two categories: traditional algorithms and ML (Machine Learning)-based algorithms.

[0003] Traditional algorithms mainly fall into four categories: graph search algorithms (such as Dijkstra's algorithm), sampling-based algorithms (such as PRM, Probabilistic Road Map), curve interpolation algorithms (such as spline curve algorithms), and reaction-based algorithms (such as PFM, Potential Field Method). ML-based control algorithms consist of classic ML algorithms (such as random forest algorithms), deep learning models such as CNN (Convolutional Neural Networks), and RL (Reinforcement Learning) algorithms (such as action-critic algorithms and proximal policy optimization algorithms).

[0004] Graph search algorithms include best-first search, depth-first search, and breadth-first search. The best-first search algorithm calculates a heuristic criterion (e.g., value or cost) for adding each node and edge in the search tree. Based on this, the search process is guided by the value gain in the search path to determine which branch to expand in the search tree. The depth-first search algorithm continuously expands the search tree in depth by prioritizing the path from the starting point to the end point until a feasible path is found. The breadth-first search algorithm is similar to the depth-first search algorithm, except that the search tree is expanded with a breadth-first strategy. However, as image resolution increases, the computational cost of graph search algorithms is high and they are insensitive to rapid environmental changes.

[0005] Sampling-based algorithms generate paths by randomly sampling a fixed workspace. To build a tree, the RRT (Rapidly-exploring Random Trees) algorithm attempts to uniformly explore the workspace through random search. The PRM (Probabilistic Roadmap Method) algorithm consists of two phases: a learning phase (preparing a collision-free probabilistic roadmap) and a query phase (constructing a trajectory between the origin and the goal based on the probabilistic roadmap). It is worth noting that sampling-based motion planning has a slower response speed and a higher collision rate due to the higher initialization delay. Due to the large search space in the application, these methods are also computationally expensive.

[0006] Curve interpolation algorithms are designed to interpolate a set of mathematical rules to create trajectories. For example, clothoids, polynomials, and splines can be used as trajectory smoothing and curve generation methods.

[0007] Unlike graph search algorithms that spend a lot of time on global path planning, reaction-based algorithms aim to react or perform local control based on the current environment. Reaction-based algorithms are widely used in industrial fields, such as potential field methods and dynamic window methods. The potential field is a variable real-valued function U, whose value can be regarded as the cost of achieving the goal (such as the distance to the target position or posture), and its gradient can be regarded as the speed. Its inverse gradient Depending on the information x of the robot at that time (such as the position of the robot), it is a vector that represents the local direction of reducing U. Therefore, the robot will be driven to move along the inverse gradient of the potential field in this type of potential field to minimize U. However, a disadvantage of methods such as PFM is that when the optimization objective function is a non-convex function, it tends to fall into local minima. In addition, when the robot moves at high speed between obstacles that are very close, it will produce violent motion oscillations. In practical applications, some of its variants, such as the VFH (Vector Field Histogram) algorithm and the HPF (Harmonic Potential Field) algorithm, can partially alleviate these shortcomings.

[0008] ML-based motion control algorithms can be divided into two categories: classical ML algorithms and RL-based algorithms. Among classical ML algorithms, CNNs have greatly improved the ability to extract high-level features from images. While they are commonly used in image-related tasks, including motion planning, they do not consider any temporal or sequential factors in the problem. Combining neural networks with LSTMs (Long Short-Term Memory) or Markov chain-based algorithms (such as Q-learning) can achieve sequential motion planning. However, these algorithms have limitations in terms of efficiency, such as slow network convergence and high training data requirements. Google DeepMind's Natural DQN (Deep Q-Learning) partially addresses this issue by reusing old data in the reply buffer to improve efficiency. To further improve robustness against biased state-action value (Q-value) estimates, Double DQN and Dueling DQN have been proposed to address issues caused by noise. Policy gradient algorithms, on the other hand, offer greater stability in network convergence, but their convergence speed is limited. To address this issue, the actor-critic algorithm was invented, which adds a judgment structure to update the state value estimate according to the improved strategy, thereby improving the convergence speed. Later, various methods such as A2C (Advantage Actor-Critic), PPO (Proximal Policy Optimization), TRPO (Trust Region Policy Optimization), and DDPG (Deep Deterministic Policy Gradient) were proposed to improve convergence stability while maintaining a faster convergence speed. However, the implementation of these learning-based algorithms still requires a lot of computing and data or environment settings. Summary of the Invention

[0009] Some embodiments of the present disclosure provide a control method for a robotic arm, the robotic arm comprising a base, at least one connecting rod, and a joint connecting the at least one connecting rod, wherein the at least one connecting rod comprises an end effector, the control method comprising:

[0010] Obtaining the target pose of the end effector in world coordinates and the forward kinematics information of the robotic arm;

[0011] determining a second current pose of the end effector in world coordinates based on a first joint angle of the robotic arm, the forward kinematics information, and a first current pose of the end effector in a coordinate system of a joint connected to the end effector;

[0012] determining a loss function, the loss function including a first loss function corresponding to a difference between the target pose and the second current pose;

[0013] Based on the loss function, the first joint angle is updated by a gradient descent algorithm to control the movement of the robotic arm.

[0014] Optionally, the target pose includes a first three-dimensional grasping point coordinate, a first grasping direction vector, and a first grasping direction normal vector;

[0015] The second current posture includes a second three-dimensional grasping point coordinate, a second grasping direction vector, and a second grasping direction normal vector.

[0016] Optionally, the first loss function includes:

[0017] A first sub-loss function, wherein the first loss function corresponds to a difference between the first sub-three-dimensional grasping point coordinates and the second three-dimensional grasping point coordinates, and the first sub-loss function corresponds to a first weight value;

[0018] a second sub-loss function, the second sub-loss function corresponding to a difference between the grasping direction vector and the second grasping direction vector, the second sub-loss function corresponding to a second weight value;

[0019] A third sub-loss function, the third sub-loss function corresponding to the difference between the grasping direction normal vector and the second grasping direction normal vector, the third sub-loss function corresponding to a third weight value.

[0020] Optionally, the control method includes:

[0021] Determining an updated second current pose of the end effector in world coordinates according to the updated first joint angle, the forward kinematics information, and the first current pose;

[0022] Obtaining a first velocity of the end effector according to the second current pose, the updated second current pose, and a time step of the gradient descent algorithm;

[0023] determining a predicted pose of the end effector in world coordinates based on the first velocity of the end effector, the updated second current pose, and a time step of the gradient descent algorithm;

[0024] Wherein, determining the loss function includes:

[0025] A fourth sub-loss function corresponding to the difference between the predicted pose and the updated second current pose and a fourth weight value corresponding to the fourth sub-loss function are determined.

[0026] Optionally, the control method includes:

[0027] Determining an updated second current pose of the end effector in world coordinates according to the updated first joint angle, the forward kinematics information, and the first current pose;

[0028] Obtaining a velocity of a joint of the robotic arm according to the second current pose, the updated second current pose, and a time step of the gradient descent algorithm;

[0029] determining a predicted joint angle of the robotic arm based on a velocity of a joint of the robotic arm, the updated first joint angle, and a time step of the gradient descent algorithm;

[0030] Determining a predicted pose of the end effector in world coordinates according to the predicted joint angle of the robotic arm, the forward kinematics information, and a first current pose of the end effector;

[0031] Wherein, determining the loss function includes:

[0032] Determine a fifth sub-loss function corresponding to the difference between the predicted pose and the updated second current pose and a fifth weight value corresponding to the fifth sub-loss function.

[0033] Optionally, the control method further includes:

[0034] Obtaining a three-dimensional model of the target object;

[0035] Based on the three-dimensional model, a preset number of balls are generated to represent the surface contours of different regions of the target object, wherein the density of the balls in the different regions corresponds to the geometric complexity of the surface contours of the different regions.

[0036] Optionally, the target object includes the robotic arm and an obstacle, and the loss function further includes a sixth sub-loss function corresponding to avoiding a collision between the robotic arm and the obstacle, and the sixth sub-loss function corresponds to a sixth weight value;

[0037] Determining the loss function further includes:

[0038] Determine all first ball pairs based on the ball of the robotic arm and the ball of the obstacle, wherein the first ball pair is composed of any ball of the obstacle and any ball of the robotic arm;

[0039] The sixth sub-loss function is determined based on a comparison between a spherical surface distance between two balls in the first ball pair and a preset threshold.

[0040] Optionally, the obstacle may include a flexible object, and obtaining the three-dimensional model of the target object includes:

[0041] Get data of real-time video stream;

[0042] A three-dimensional model of the target object is generated according to the two-dimensional image of the flexible object in the video stream.

[0043] The loss function further includes a seventh sub-loss function corresponding to avoiding a collision between the robotic arm and the flexible object, and the seventh sub-loss function corresponds to a seventh weight value;

[0044] Determining the loss function further includes:

[0045] determining all second ball pairs based on the ball of the robotic arm and the ball of the flexible object, wherein the second ball pair is composed of any ball of the robotic arm and any ball of the flexible object;

[0046] The seventh sub-loss function is determined based on a comparison between a spherical surface distance between two balls in the second ball pair and a preset threshold.

[0047] Optionally, the target object includes the robotic arm, and the loss function further includes an eighth sub-loss function corresponding to preventing links of the robotic arm from colliding with each other, and the eighth sub-loss function corresponds to an eighth weight value;

[0048] Determining the loss function includes:

[0049] Based on the balls of the robotic arm, determining all third ball pairs, wherein the third ball pairs are composed of any two balls on different links of the robotic arm;

[0050] The eighth sub-loss function is determined based on a comparison between a spherical surface distance between two balls in the third ball pair and a preset threshold.

[0051] Optionally, the target object includes the robotic arm and a physical surface within the workspace of the robotic arm, and the loss function further includes a ninth loss function corresponding to avoiding collision between the robotic arm and the physical surface, wherein the ninth sub-loss function corresponds to a ninth weight value. Determining the loss function includes: determining all fourth ball pairs based on the balls of the robotic arm and the balls on the physical surface, wherein the fourth ball pair is composed of any ball of the robotic arm and any ball on the physical surface; and determining the ninth sub-loss function based on a comparison of a spherical distance between two balls in the fourth ball pair and a preset threshold.

[0052] Optionally, the loss function further includes a tenth sub-loss function for limiting the angle range of the joint angle of the robotic arm, and the tenth sub-loss function corresponds to a tenth weight value;

[0053] The determining of the loss function includes determining the tenth sub-loss function based on a comparison between the first joint angle of the robotic arm and a preset joint angle threshold.

[0054] Optionally, the loss function further includes an eleventh sub-loss function for limiting the speed of the end effector, and the eleventh sub-loss function corresponds to an eleventh weight value;

[0055] Determining the loss function includes:

[0056] According to the updated first joint angle, the forward kinematics information and the first current pose, the updated second current pose of the end effector in the world coordinates is determined; according to the second current pose, the updated second current pose and the time step of the gradient descent algorithm, the second speed of the end effector is obtained; based on the comparison between the second speed of the end effector and a preset speed threshold, the eleventh sub-loss function is determined, wherein the preset speed threshold includes a maximum speed threshold, and optionally, the preset speed threshold may also include a minimum speed threshold.

[0057] Optionally, when the value of each sub-loss function of the loss function is lower than a predetermined threshold, the current posture of the robotic arm matches the target posture.

[0058] Optionally, the control method further includes: when the value of each sub-loss function of the loss function is lower than a predetermined threshold, receiving a first instruction, wherein the first instruction instructs the end effector to perform an object grasping operation.

[0059] Optionally, the control method further includes: determining the state information of the robotic arm based on the control instructions received by the robotic arm, the target object and whether the end effector executes grasping the object; and adjusting the number, type and corresponding weight values of the sub-loss functions in the loss function based on the state information of the robotic arm.

[0060] Optionally, determining the loss function includes: obtaining state information of the robotic arm, wherein the state information indicates at least one of the following: whether the robotic arm is holding an object; whether the robotic arm is moving; whether the robotic arm is stationary; whether the current posture of the robotic arm matches the target posture; whether the robotic arm is grasping an object; whether the robotic arm is releasing the held object; and adjusting the weight value corresponding to each sub-loss function of the loss function according to the state information; optionally, determining the loss function may include: determining the state information according to the instructions received by the robotic arm and whether the end effector executes grasping the object.

[0061] Optionally, the control method further includes: when the end effector holds the object, stacking a ball representing the surface contour of the end effector and a ball representing the object, wherein the stacked balls represent the object as an extension of the end effector to represent the surface contour of the extended end effector.

[0062] Optionally, the end effector is provided with a sensor, for example, a tactile sensor, and the control method further includes:

[0063] receiving image information of all objects in the working space of the robotic arm and the robotic arm sent by an image acquisition module;

[0064] Obtaining feedback information sent by the sensor;

[0065] It is determined whether the end effector holds an object according to the image information and the feedback information.

[0066] Optionally, the control method further includes:

[0067] receiving image information of all objects in the working space of the robotic arm and the robotic arm sent by an image acquisition module;

[0068] Displaying the image information on a graphical user interface of a display; and / or,

[0069] The instruction is generated in response to a user operation on the graphical user interface.

[0070] Some embodiments of the present disclosure provide a control system for a robotic arm, the control system comprising: one or more processors, the one or more processors being configured to execute a computer program to control the robotic arm to implement the aforementioned control method.

[0071] Optionally, the control system further includes: an image acquisition module, which is used to acquire image information of all objects in the workspace and the robotic arm, and send the image information to the one or more processors.

[0072] Optionally, the control system further includes:

[0073] A robotic arm, wherein the robotic arm comprises a base, at least one link, and a joint connecting the at least one link, wherein the at least one link comprises an end effector;

[0074] a sensor provided on the end effector of the robotic arm, for example, a tactile sensor;

[0075] The one or more processors are further configured to:

[0076] receiving image information of all objects in the working space of the robotic arm and the robotic arm sent by an image acquisition module;

[0077] Obtaining feedback information sent by the sensor;

[0078] It is determined whether the end effector holds an object according to the image information and the feedback information.

[0079] Optionally, the control system further comprises: a display, wherein the display is configured to: display the image information processed by the one or more processors on the user interface; and / or receive user operations through the user interface;

[0080] The one or more processors are configured to generate the instructions based on the operations.

[0081] Some embodiments of the present disclosure provide an electronic device, including a memory, one or more processors, and a computer program stored in the memory and used to run on the processors, wherein the one or more processors implement the aforementioned method for controlling the robotic arm when executing the computer program.

[0082] Some embodiments of the present disclosure provide a computer-readable storage medium having a computer program stored thereon, which implements the aforementioned method for controlling the robotic arm when executed by one or more processors.

[0083] Some embodiments of the present disclosure provide a computer program product, including a computer program, which implements the aforementioned method for controlling a robotic arm when executed by one or more processors.

[0084] Some embodiments of the present disclosure achieve control of a robotic arm by using a gradient descent algorithm to iteratively update the joint angles of the robotic arm using a loss function that includes the difference between the target and current poses of the robotic arm in the same coordinate system. This control method can generate an optimized trajectory for the robotic arm's end effector, guiding it toward the target pose without requiring pre-training data, thus reducing computational costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] To more clearly illustrate the technical solutions of the embodiments of this specification, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are merely examples or embodiments of this specification. For those skilled in the art, it is possible to apply this specification to other similar scenarios based on these drawings without inventive effort.

[0086] Figure 1 A flowchart of a method for controlling a robotic arm provided in Example 1 of the present disclosure.

[0087] Figure 2 A schematic diagram of a simulated robotic arm provided in some embodiments of the present disclosure is shown.

[0088] Figure 3 A schematic diagram of a physical robotic arm provided in some embodiments of the present disclosure is shown.

[0089] Figure 4 Shows a Model Predictive Control Schematic diagram of the algorithm principle.

[0090] Figure 5a A flow chart of a method for controlling a robotic arm provided in some embodiments of the present disclosure is shown.

[0091] Figure 5b A flow chart of a method for determining a loss function provided in some embodiments of the present disclosure is shown.

[0092] Figure 5c A flow chart of a method for controlling a robotic arm provided in some embodiments of the present disclosure is shown.

[0093] Figure 5d A flow chart of a method for determining a loss function provided in some embodiments of the present disclosure is shown.

[0094] Figure 6 A method of using a sphere to represent the surface contours of different areas of the target object is shown in some embodiments of the present disclosure.

[0095] Figure 7 A schematic diagram of a neighborhood provided in some embodiments of the present disclosure is shown.

[0096] Figure 8 An example of constructing a three-dimensional hand model from a camera image in real time, provided in some embodiments of the present disclosure, is shown.

[0097] Figure 9 A diagram showing some examples using collision sphere representation.

[0098] Figure 10 A schematic diagram showing a process of a gripper at the end of a robot grasping a target object provided in some embodiments of the present disclosure.

[0099] Figure 11 A flow chart of a method for determining a loss function provided in some embodiments of the present disclosure is shown.

[0100] Figure 12 A schematic diagram showing the conversion relationship between states of a robotic arm provided in some embodiments of the present disclosure.

[0101] Figure 13 A schematic diagram showing the interaction process between a robotic arm and an object during a grasping-and-placing operation provided in some embodiments of the present disclosure is shown.

[0102] Figure 14 A graphical user interface for a user to control a robotic arm provided in some embodiments of the present disclosure is shown.

[0103] Figure 15 A schematic diagram of a control system of a robotic arm provided in some embodiments of the present disclosure is shown.

[0104] Figure 16 A schematic diagram of a multi-threaded system structure provided in some embodiments of the present disclosure is shown.

[0105] Figure 17 A schematic structural diagram of an electronic device provided in some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0106] The present invention is further described below by way of examples, but the present invention is not limited to the scope of the examples.

[0107] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various places herein does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0108] It should be understood that the terms "device," "system," "unit," and / or "module" used herein are a method for distinguishing different components, elements, parts, portions, or assemblies at different levels. However, if other terms can achieve the same purpose, the terms may be replaced by other expressions.

[0109] As used herein, unless the context clearly indicates otherwise, the terms "a," "an," "an," and / or "the" are not intended to refer to the singular but may include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements specifically identified, and these steps and elements do not constitute an exclusive list. A method or apparatus may also include additional steps or elements.

[0110] The definition of inclusion herein, such as the terms “having”, “may have”, “include” or “may include” as used herein, indicates the existence of the corresponding functions, operations, elements, etc. herein, and does not limit the existence of one or more other functions, operations, elements, etc. In addition, it should be understood that the terms “including” or “having” as used herein indicate the existence of the features, numbers, steps, operations, elements, components or their combination described in the specification, and do not exclude the existence or addition of one or more other features, numbers, steps, operations, elements, components or their combination.

[0111] Flowcharts are used herein to illustrate the operations performed by the systems according to the embodiments of the present invention. It should be understood that the preceding or following operations do not necessarily need to be performed in exact order. Instead, the steps may be processed in reverse order or simultaneously. Furthermore, other operations may be added to these processes, or one or more operations may be removed from these processes.

[0112] In some examples of this disclosure, the following method steps can be implemented by a program or a custom circuit, or a combination of a custom circuit and a program. For example, the following method steps can be performed by a combination of one or more GPUs, one or more CPUs, or any other technically feasible one or more processors such as an ASIC chip. In some examples of this disclosure, one or more processors such as an ASIC chip may include a memory. Further, those skilled in the art will understand that the scope of protection of this disclosure may include any system and device that can perform the following method steps.

[0113] Example 1

[0114] Some examples of this disclosure provide a method for controlling a robotic arm. In some examples of this disclosure, the method can be used to plan the motion trajectory of the robotic arm.

[0115] In some examples of this disclosure, the robot may be a robotic arm. The robotic arm includes a base, at least one connecting rod, and a joint connecting the at least one connecting rod, wherein at least one connecting rod includes an end effector. The joint can generate translational and rotational motions for the robotic arm, thereby enabling the end of the robotic arm to reach a desired position.

[0116] like Figure 1 As shown, the control method includes:

[0117] S110 , obtaining the target pose of the end effector in world coordinates and the forward kinematics information of the robotic arm.

[0118] In some examples disclosed herein, the origin of the robot base coordinate system is the same as the origin of the world coordinate system, that is, the target pose is the pose of the end effector in the robot base coordinate system.

[0119] In some examples of this disclosure, the target pose can be represented by three vector parameters: and in, is the vector representing the position point of the target pose, is the vector representing the direction of the target pose, A person skilled in the art will appreciate that the target posture is not limited to being represented by the above three vector parameters, but may also be represented by other means, such as a matrix.

[0120] Forward kinematics

[0121] Forward kinematics is a series of physical transformations that describe the physical transformations available to each component in the robot. The physical pose of a robot component is determined not only by the joint angles or translation phases of its direct connections (relative pose with respect to its neighboring components), but also by the relative pose of that neighboring component with respect to its own neighbors. Therefore, it is more straightforward to describe the physical transformation of a robot component as a series of sequential transformations. In computer graphics, this composite transformation can be simply described as the product of the component's geometric point cloud coordinate grid and a 4x4 matrix. A unique 4x4 matrix is usually provided for each independent component of the robot to describe the complete forward kinematics.

[0122] In forward kinematics, each joint in a robotic arm has a coordinate system. For example, the coordinate systems are set as follows: for each coordinate system, the z-axis is set to coincide with the axis of the joint, the x-axis points in the direction of the next joint and is perpendicular to the z-axis of the coordinate system corresponding to the previous joint and the z-axis of the current coordinate system. The direction of the y-axis is determined by the right-hand rule. Rotation and translation occur between adjacent coordinate systems, and the coordinate changes between adjacent links can be described using DH (Denavit-Hartenberg) parameters.

[0123] DH parameters consist of four basic elements: θ (Theta), d (Distance), a (Link Length), and α (Twist Angle); θ represents the rotation angle around the previous z-axis, d represents the translation distance along the previous z-axis, a represents the length of the common perpendicular from the previous z-axis to the current z-axis, and α represents the angle between the two z-axes. The DH parameter table uses a series of values for these four parameters to define the position and orientation of each joint of the robot, thereby describing the posture of the entire manipulator. For example, a planar manipulator with two joints might have a DH parameter table containing two rows, each containing four parameter values. These values can be used to determine the position and posture of each joint relative to the previous joint, ultimately determining the position and posture of the end effector. The end effector can be, for example, a gripper.

[0124] Forward kinematics information can be obtained, for example, by inputting a URDF (United Robotics Description Format) file.

[0125] The forward kinematics information can be used to transform the pose from the end effector coordinate system to the robot base coordinate system.

[0126] URDF file

[0127] A Unified Robot Description Format (URDF) file is an example of a data format used to describe the physical geometry of a robot and its components as point cloud meshes, as well as the robot's degrees of freedom to implement its physical structure transformations, such as joint rotations and physical translations. Mesh components in a URDF file may include a visual mesh (a large number of 3D polygons for visualization) and a collision mesh (a smaller number of 3D polygons for efficient collision detection).

[0128] S120 , determining a second current pose of the end effector in world coordinates based on the first joint angle of the robotic arm, forward kinematics information, and a first current pose of the end effector in a coordinate system of a joint connected to the end effector.

[0129] The first current posture may be a posture of the end effector in a coordinate system of a joint connected to the end effector. In one example, the end effector may point to the Z axis. Figure 2 A simulated robotic arm 210 is shown, including an end effector 214. A first current pose of the end effector 214 can be represented using three vector parameters: and in, is the position vector representing the first current pose, is the vector representing the direction of the first current pose, is the vector representing the normal direction of the first current pose. Based on these vector parameters, the first current pose in the coordinate system of the last joint of the manipulator can be transformed into the second current pose in the coordinate system of the robot base, which can be used to evaluate the alignment of the end effector pose with the target pose.

[0130] The following can be explained with a specific example and meaning.

[0131] Indicates that the end effector is located at a point 15 cm away from the coordinate origin along the positive direction of the z-axis;

[0132] Indicates that the direction of the end effector is along the z-axis;

[0133] Or [-1,0,0], indicating that the normal direction relative to the end effector is in the positive direction of the x-axis or the negative direction of the x-axis.

[0134] Figure 3 An example of a physical robotic arm 230 is shown, which includes an end effector 234. When the robot's task is to grasp a target object 238, the target pose is the pose required for the end effector 234 to grasp the target object 238, that is, the target grasping pose. The target grasping pose can also be represented by three vector parameters: and in, is the vector representing the position point of the target grasping posture, is the vector representing the direction of the target grasping pose, is the vector representing the normal direction of the target pose

[0135] In order to accurately grasp the target object 238, the target grasping pose relative to the target object must first be determined for the end effector 234. The target grasping pose relative to the target object 238 provides the relative position and orientation of the end effector in the coordinate system of the target object. For example, the target grasping pose relative to the target object can be directly obtained through user input or obtained through a grasping pose generation algorithm. The grasping pose generation algorithm can be obtained by a 4×4 transformation matrix P G2O Indicates. G2O Convert the end-effector's rest pose to the desired transformation in the target object's coordinate system.

[0136] Assuming that the base of the robot is located at the origin of the world coordinate system, the world coordinate system coincides with the robot base coordinate system. After the pose is converted to the world coordinate system, there is no need to convert it from the world coordinate system to the robot base coordinate system. Therefore, when obtaining the 6D pose of the target object, After that, the target grasping pose in the target object's coordinate system can be converted to the world coordinate system, thereby obtaining the target grasping pose in the world coordinate system. The specific formula is as follows:

[0137]

[0138] in, is the vector of the position point representing the target grasping posture; is a vector representing the direction of the target grasping posture; is the vector representing the normal direction of the target pose; is the vector representing the position point of the first current pose; is a vector representing the direction of the first current pose; is the vector representing the normal direction of the first current pose; P 6D is the 6D pose of the target object; P G2O is the transformation matrix.

[0139] Get the target grasping pose in the world coordinate system and The end effector's position can be controlled to gradually approach the target grasping position. Given the initial value of the robot's joint angle θ ′ and target grasping pose and After that, the goal is to generate a smooth joint angle trajectory Tj = {θ n |n∈{1,2,…N}} can be implemented by the robot; where N is the number of steps.

[0140] In order to guide the end effector toward the target pose, the forward kinematic transformation matrix FK(θ) is used to transform and Convert to the robot base coordinate system to obtain the current pose of the end effector in the robot base coordinate system. The conversion formula is as follows:

[0141]

[0142] in, is the vector representing the position point of the second current pose; is the vector representing the direction of the second current pose; is the vector representing the normal direction of the second current pose; is the vector representing the position point of the first current pose; is a vector representing the direction of the first current pose; is the vector representing the normal direction of the first current pose; FK(θ) is the forward kinematics function with θ as the variable.

[0143] S130. Determine a loss function, which includes a first loss function corresponding to the difference between the target posture and the second current posture. The loss function can also be understood as a cost function. In some examples disclosed herein, the first loss function can correspond to the deviation (i.e., the difference) between the current posture and the target posture in position and orientation. By using the information of the differentiable forward kinematics of the manipulator and the gradient descent algorithm, the deviation between the current posture of the end effector and the target posture in position and orientation can be minimized in an iterative manner, so that a set of joint angles of the manipulator can be calculated, so that the manipulator can move smoothly to the target posture.

[0144] S140. Based on the objective function, update the first joint angle by a gradient descent algorithm to control the movement of the robotic arm.

[0145] In some examples of this disclosure, since the forward kinematics function FK(θ) is differentiable, the objective function can be minimized by gradient descent. That is, the value of θ can be adjusted iteratively to reduce the loss. The update rule for θ at the tth iteration is defined by the following formula:

[0146]

[0147] Among them, θ t is the joint angle at time step t; k t is the step size for each θ, which can be determined according to the optimization algorithm used; θ t+1 is the joint angle at time step t+1; is the gradient of the objective function Loss relative to the parameter θ at the tth iteration. For example, k t This can be a scalar value used in PyTorch optimization algorithms.

[0148] This optimization process is continuously iterated, and each time the joint angle θ is updated, a smooth trajectory is generated, which gradually reduces the deviation between the current position and direction of the end effector and the target position. In some examples disclosed herein, when the value of the objective function is lower than a predetermined threshold, it can be considered that the position of the end effector is close to the target position, and the iteration can be terminated at this time. Therefore, in some examples disclosed herein, by controlling the robotic arm to run according to the calculated trajectory, it is possible to achieve the end effector reaching the target position, for example, for Figure 2 The simulated robotic arm 210 shown can enable the end effector 214 to reach a target posture.

[0149] Loss Function

[0150] In some examples of this disclosure, the loss function can be composed of one or more sub-loss functions and weight coefficients corresponding to each sub-loss function. The weight coefficient can also be understood as a weight value. In some examples of this disclosure, the sub-loss function includes a first loss function, which represents the difference between the second current pose and the target pose.

[0151] For example, individual sub-loss functions are combined with corresponding weight coefficients to form the overall loss function, as shown below:

[0152] Loss=λ1L1+λ2L2+…+λ i L i

[0153] Among them, Loss is the loss function; L1 is the i-th sub-loss function; λ i is the weight coefficient corresponding to the ith sub-loss function. For example, the weight coefficient of each sub-loss function can be defined in a pre-set manner. In some examples disclosed herein, each weight coefficient can also be further adjusted.

[0154] In some examples disclosed herein, the first loss function includes at least one of a position sub-loss function, a direction sub-loss function, and a normal direction sub-loss function. The position sub-loss function represents the error between the second current pose and the target pose in the position point, the direction sub-loss function represents the error between the second current pose and the target pose in the direction vector, and the normal direction sub-loss function represents the error between the second current pose and the target pose in the normal direction.

[0155] In some examples disclosed herein, the position point loss function can use the square of the Euclidean distance to represent the difference between the current position point of the end effector and the position point of the target pose. For example, the formula is as follows:

[0156]

[0157] Among them, L dist is the position point objective function, is the position point of the current pose of the end effector; The inventors found that the square of the Euclidean distance can be used to achieve an intuitive distance measurement in 3D space, and it is differentiable.

[0158] In some examples disclosed herein, the direction sub-loss function uses cosine similarity to represent the difference between the direction of the end effector's current pose and the direction of the target pose. For example, the formula is as follows:

[0159]

[0160] Among them, L dir is the direction objective function; is a vector representing the direction of the current pose of the end effector; The inventors found that the cosine similarity can achieve an intuitive vector angle difference measurement in 3D space, and it is differentiable.

[0161] In some examples disclosed herein, the normal direction sub-loss function is used to describe the difference between the normal direction of the end effector's current pose and the normal direction of the target pose. If the end effector is symmetrical, the absolute value of the cosine similarity can be used to constrain this. For example, the formula is as follows:

[0162]

[0163] Among them, L n_dir is the normal direction objective function; is the vector representing the normal direction of the current pose of the end effector; The inventors found that the absolute value of the cosine similarity can be used flexibly to align the normal vector with the target vector or to align it in the opposite direction.

[0164] In some examples of the present disclosure, the sub-loss function may further include at least one of a speed limit loss function and an angle limit loss function. The angle limit objective function may be used to limit the angle range of the end effector, while the speed limit objective function may be used to limit the speed range of the end effector.

[0165] Including the angle constraint loss function in the loss function helps ensure that the robot operates within the mechanical limitations of the manipulator, thereby avoiding possible damage or instability of the manipulator during movement.

[0166] For example, the angle-constrained loss function can be expressed as follows:

[0167]

[0168] Among them, L angle-limit (a) is the angle constraint objective function; a is the joint angle in radians; a low is the lower limit of a; a high is the upper limit of a; C is a scaling constant that determines the severity of the penalty.

[0169] The speed limit loss function is used to penalize the link that exceeds the specified speed range limit, thereby limiting the speed of the end effector during the movement. Including the speed limit loss function in the loss function can have the following benefits:

[0170] 1. Limiting the maximum speed helps improve the safety of the robot arm during movement.

[0171] The end effector is usually the fastest moving part of a robotic arm. Limiting the physical speed of the end effector in the gradient descent-based trajectory planning method helps improve the safety of human-machine interaction.

[0172] 2. Limiting the minimum speed helps to escape the local minimum of the loss function.

[0173] While the robot's trajectory planning method in this embodiment can achieve low-level goals while satisfying multiple constraints, optimization-based methods are still prone to falling into local minima of the loss function, especially when the robot's environment has complex obstacles, such as large obstacles with large aspect ratios. In such cases, a lower end-effector velocity limit may force the robot to continue moving, eventually escaping the local minimum of the loss function while also avoiding collisions between the robot and the obstacles in the environment.

[0174] For example, suppose the forward kinematic function corresponding to the end effector is FK ee (θ t ), and it takes time Δt between consecutive robot control command executions, then the speed limit loss function can be expressed by the following formula:

[0175]

[0176] Among them, FK ee (θ t ) is the forward kinematic function of the end effector; is the vector representing the position point of the first current pose; L speed-limit (v ee,t ) is the speed limit objective function; v lim,min (t) is the lower limit of velocity, which is a value that changes with time; is the position of the end effector at time step t; v eef,t is the speed of the terminal execution at time step t; V max is the upper limit of the speed, which is a constant value; C is a scaling constant that determines the severity of the penalty.

[0177] In some examples of this disclosure, v lim,min(t) is a curve that changes with time, which may start from a positive value of, for example, 10 cm / s, and continuously decrease to 0 cm / s over time.

[0178] Each sub-loss function in the loss function penalizes deviations from the desired pose, orientation, position, and spatial constraints, such as joint angle limits. Using these loss functions, θ is updated to generate the robot's trajectory using the gradient descent method. This process continues until the loss function falls below a predetermined threshold, indicating that the end-effector's current pose is very close to the target pose.

[0179] Model Predictive Control (MPC) algorithm

[0180] The MPC algorithm is a powerful control technique that optimizes future control actions within a finite time horizon, taking into account the system dynamics and any constraints. This approach has gained widespread application in robotics due to its ability to effectively handle multivariable systems, nonlinearities, and constraints.

[0181] The MPC algorithm optimizes the control input at each time step by solving the following optimization problem:

[0182]

[0183] Where t is the current time step; is the predicted output at time step t+k; w(k) is the reference trajectory; u(k) is the control input at time step t+k; Q and R are weight matrices; N is the prediction range; and M is the control range.

[0184] The principle of MPC algorithm is as follows Figure 4 As shown, the optimization problem minimizes the error between the predicted state and the reference state while keeping the control input within the feasible range.

[0185] The inventors found that the limitations of the MPC algorithm include:

[0186] Computational complexity: Solving the optimization problem at each time step can be computationally expensive, especially for large systems with many degrees of freedom.

[0187] Model accuracy: Performance depends on the accuracy of the system model; any error between the model and the actual robot behavior can lead to suboptimal control actions.

[0188] Real-time limitations: Real-time applications require fast computation, which may limit the complexity of the model and the prediction range.

[0189] The inventors also discovered that the advantages of the MPC algorithm include:

[0190] Optimality: MPC optimizes control over a limited range, ensuring the best performance within the model and constraints.

[0191] Handling constraints: This naturally includes restrictions on the system state and inputs, such as joint limits, velocity limits, and safety limits.

[0192] Predictive: MPC predicts future system states and is therefore well suited for dynamic and uncertain environments.

[0193] Loss function based on MPC algorithm

[0194] Predicting joint angles using joint velocities, forward and inverse kinematics (IK-MPC)

[0195] The inventors found that the MPC algorithm can optimize the joint velocity at each time step by solving the following optimization problem:

[0196]

[0197] Where t is the current time step, q(k) is the predicted joint angle at time step t+k; q ik (k) is the reference joint angle solved based on inverse kinematics; Δq(k) is the control signal, which is the joint velocity at time step t+k in this case; q min and q max is the joint angle limit, which is determined by the physical capabilities of the robot; Δq min and Δq max Enter limits for the control, i.e. the minimum and maximum allowed joint velocities.

[0198] The inventors found that the main difference between this algorithm and the joint angle prediction-based algorithm is that the inverse kinematics solution is used as a reference for the MPC algorithm. In this algorithm, the optimization can be divided into two sub-problems:

[0199] (1) MPC is used as feedforward to optimize joint velocities using inverse kinematics to guide the robot from the starting joint angle to the target angle configuration.

[0200] (2) Using differentiable forward kinematics based on gradient optimization, the joint angles predicted by MPC are added to the optimized joint angles. The sum of the two control signals makes the robot move faster.

[0201] The advantages of this algorithm are as follows:

[0202] Separate control: MPC does not interfere with gradient-based feedback optimization, and the two optimizations proceed simultaneously.

[0203] Combined speed control: MPC can limit the control signal according to user needs.

[0204] The inventors discovered that MPC can be viewed as a method for pre-planning trajectories within a certain prediction range, while ignoring potential collisions with objects in the system. This makes MPC unsuitable for real-time reactive control with multiple constraints (collisions) in a dynamic environment. However, the step size generated by MPC can be combined with gradient control based on a loss function to further optimize the robot's motion trajectory. Assuming that MPC generates a set of u(0) inputs, this input value can be used as a base value and modified by the above-mentioned gradient control-based loss function. The specific formula is as follows:

[0205] θ t =q(0)+α t

[0206]

[0207] Among them, k t is the step size for each joint angle, determined by the optimization algorithm used; is the gradient of the loss function Loss at time step t with respect to the parameter α.

[0208] In some examples of the present disclosure, after considering the principles, advantages and disadvantages of the above two MPC algorithms, the inventors found that the following two algorithms can be used to further improve the gradient-based robotic arm joint angle optimization process of the present disclosure using predictions based on future horizons.

[0209] 1. Use coordinate velocity to predict the future position of the end effector

[0210] In this algorithm, the velocity of the end effector in the previous time step can be used to predict its position in the next few time steps. Figure 5a As shown, in some examples of the present disclosure, a method 100 for controlling a robotic arm may include:

[0211] S252: Determine the updated second current pose of the end effector in world coordinates. As described above, the pose of the end effector at time step t in world coordinates, i.e., the updated second current pose, can be determined based on the updated first joint angle, forward kinematics information, and the first current pose. The position p(t) of the end effector at time step t can be determined from the pose of the end effector at time step t.

[0212] S254. Obtain the first velocity of the end effector. For example, the first velocity is the velocity of the end effector at the previous time step. As described above, the second current pose at the previous time step, the pose at time step t (i.e., the updated second current pose), and the time step k of the gradient descent algorithm can be used to obtain the first velocity of the end effector. t , obtain the first velocity of the end effector.

[0213] S256. Determine the predicted pose of the end effector in the world coordinates. For example, based on the first velocity of the end effector, the updated second current pose, and the time step of the gradient descent algorithm, the predicted pose of the end effector in the world coordinates, that is, the pose at the next time step, can be determined. Assuming that the position of the end effector at time step t is p(t) and the velocity of the end effector at time step t is v(t), a simple first-order approximation algorithm can be used to predict the position of the end effector at the next time step t+1. For example, the formula can be as follows:

[0214] p(t+1)=p(t)+v(t)Δt

[0215] Where p(t) is the position of the end effector at time step t; v(t) is the velocity of the end effector at time step t; and Δt is the time step. The Δt time step can be equal to the time step of the gradient descent algorithm. It can also be different from the time step of the gradient descent algorithm. For example, the Δt time step can be equal to several time steps of the gradient descent algorithm.

[0216] like Figure 5b As shown, determining the loss function S130 may include:

[0217] S260. Determine a fourth sub-loss function corresponding to the difference between the predicted pose and the updated second current pose and a fourth weight value corresponding to the fourth sub-loss function.

[0218] By including the fourth sub-loss function and its weight value in the loss function, it is possible to use these predicted positions to optimize the joint angle q(t) so that the end effector reaches the desired position. At the same time, the loss function can also include other sub-loss functions such as various collision sub-loss functions, so as to take into account the potential collision risks of objects in the system.

[0219] For example, an exemplary code of the algorithm may be as follows:

[0220]

[0221]

[0222] The advantages of this algorithm are as follows:

[0223] Simple: The prediction uses a first-order approximation of the velocity and is therefore computationally efficient.

[0224] Fast computation: This algorithm does not require complex robot models and can quickly recalculate future states.

[0225] 2. Predict joint angles using joint velocities and forward kinematics

[0226] In this algorithm, the joint velocity of the previous time step is used to predict the joint angle of the next few time steps. Figure 5c As shown, in some examples of the present disclosure, a method 100 for controlling a robotic arm may include:

[0227] S272. Determine the updated second current pose of the end effector in the world coordinates. As described above, the pose of the end effector at time step t in the world coordinates, i.e., the updated second current pose, can be determined based on the updated first joint angle, the forward kinematics information, and the first current pose.

[0228] S274: Obtaining the velocities of the joints of the robotic arm. Based on the second current posture, the updated second current posture, and the time step of the gradient descent algorithm, the velocities of the joints of the robotic arm, that is, the joint velocities at the previous time step, can be obtained.

[0229] S276, determine the predicted joint angle of the robot arm. For example, the configuration of the robot is determined by the joint angle q(t) and the joint velocity v q (t) description, and the prediction of joint angle is based on velocity, for example, the formula is as follows:

[0230] q(t+1)=q(t)+v q (t)Δt

[0231] Where q(t) is the joint angle at time step t; v q (t) is the joint velocity at time step t. The time step Δt can be equal to the time step k of the gradient descent algorithm t The Δt time step can also be equal to a number of gradient descent time steps k t , which is a few time steps. Therefore, based on the joint velocity of the robot at the previous time step t-1, the joint angle q(t) at the current time step t (that is, the first joint angle after update) and the time step k of the gradient descent algorithm t , the predicted joint angles of the robotic arm can be determined.

[0232] S278: Determine the predicted pose of the end effector in world coordinates. As described above, the predicted pose of the end effector in world coordinates can be determined based on the predicted joint angles of the manipulator, the forward kinematics information, and the first current pose of the end effector.

[0233] like Figure 5b As shown, determining the loss function S130 may include:

[0234] S280: Determine a fifth sub-loss function corresponding to the difference between the predicted pose and the updated second current pose and a fifth weight value corresponding to the fifth sub-loss function.

[0235] By including the fifth sub-loss function and its weight value in the loss function, the joint angle q(t) can be optimized by predicting the next joint configuration and minimizing the loss function. At the same time, the loss function can also include the errors in the position and control force of the end effector and other sub-loss functions such as various collision sub-loss functions, thereby taking into account the potential collision risks of objects in the system.

[0236] For example, an exemplary code of the algorithm may be as follows:

[0237]

[0238]

[0239] The advantages of this algorithm are as follows:

[0240] Direct control: This algorithm directly controls the robot's joint angles, usually as input to the robot's actuators.

[0241] Versatility: The algorithm can be used for both position and force control tasks.

[0242] Based on the principles of MPC, the predictive algorithms presented above provide a robust framework for controlling digital robots, particularly in dynamic and constrained environments. By combining predictions of either the end-effector position or joint angles, the system can predict future states and optimize control inputs accordingly. Predicting the end-effector position offers a simpler and more computationally efficient approach but is limited by the constant velocity assumption. Predicting joint angles offers more functionality and allows for direct control of the robot's actuators, but requires careful management of velocity estimation errors.

[0243] A sphere representing the surface outline of an object

[0244] The inventors discovered that while using simplified geometric primitives (such as spheres) to represent three-dimensional surfaces has been used in applications requiring collision detection, conventional techniques rely on uniform sampling or simple geometric approximations. These techniques lack adaptability to local surface features such as curvature and vertex density that represent the geometric complexity of the surface contour. This results in poor accuracy or excessive computational overhead when modeling complex geometries.

[0245] The inventors have discovered that if a preset number of balls are generated based on a three-dimensional model to represent the surface contours of different regions of the target object, and the density of the balls in different regions corresponds to the geometric complexity of the surface contours of different regions, it is possible to accurately model complex geometric bodies while also balancing the computational overhead. Figure 6As shown, some examples of the present disclosure provide a method 400 for using balls to represent the surface contours of different areas of the target object, which includes: step 410, obtaining a three-dimensional model of the target object; step 420, based on the three-dimensional model, generating a preset number of balls to represent the surface contours of different areas of the target object, wherein the density of the balls in the different areas corresponds to the geometric complexity of the surface contours of the different areas.

[0246] For example, different regions include a first region and a second region, and the geometric complexity of the surface contour of the first region is greater than the geometric complexity of the surface contour of the second region; the density of balls in different regions corresponds to the geometric complexity of the surface contour of different regions, including: the density of balls in the first region is greater than the density of balls in the second region.

[0247] Optionally, in order to determine the density weight values of candidate sampling points in different regions of the target object, where the density weight values represent the geometric complexity of the surface contours of different regions, the following methods may be used:

[0248] a) Determine a neighborhood of the candidate sampling point based on the position of the candidate sampling point and a preset neighborhood size value.

[0249] b) Determine the density weight of the candidate sampling point based on the average distance between the candidate sampling point and other candidate sampling points in the neighborhood. Specifically, the density weight of the candidate sampling point can be determined according to the following formula:

[0250]

[0251] Among them, w i is the density weight; is the average distance; ∈ is a constant.

[0252] Specifically, ∈ is a small constant that prevents the denominator of the above formula from being 0, resulting in w i The numerical value of is unstable. From the above formula, we can see that w i along with decreases and increases, because represents the average distance between the candidate sampling point and other candidate sampling points in the neighborhood, The distribution density of candidate sampling points in the neighborhood increases, so the above formula assigns a higher density weight to the area of candidate sampling points, ensuring that when determining sampling points from the candidate sampling points, priority is given to determining sampling points in areas with high geometric complexity of the surface contour.

[0253] The average distance can be determined by the following formula

[0254]

[0255] in, is the average distance, is the set of other vertices in the neighborhood, p i is the candidate sampling point as the neighborhood center; p j are other candidate sampling points in the neighborhood.

[0256] Optionally, the three-dimensional model is a polygonal mesh model, wherein the polygonal mesh model is composed of a set of vertices, edges and faces, and the candidate sampling points are vertices of the polygonal mesh model.

[0257] Optionally, the neighborhood of the candidate sampling point can be determined by taking the candidate sampling point as the center and determining a range of polygons with a number of edges not greater than n between the candidate sampling point and the candidate sampling point as the neighborhood of the candidate sampling point, where the preset neighborhood size value is n, and n is a positive integer.

[0258] Computer Graphics

[0259] Computer graphics is the field of computer-based image rendering techniques that generate photo-realistic two-dimensional images. In this field, three-dimensional objects are represented using a three-dimensional object model, which is typically composed of a point cloud triangulated mesh represented by the coordinates of vertices and triangular faces, in addition to a color texture map of the model. A virtual camera and lighting model are used to mathematically render the three-dimensional object into a corresponding 2D image. The rendering methods themselves range from computationally inexpensive methods (such as rasterization) to computationally expensive methods (such as ray tracing). In some examples of the present disclosure, rasterization-based image rendering techniques can be used.

[0260] Specifically, in computer graphics, polygonal mesh models define the shape and outline of each 3D object. Polygonal meshes are built from smaller interconnected planes (usually triangles or rectangles) that act like a 3D puzzle. Figure 1 Each vertex in a polygon mesh stores x, y, and z coordinate information. Each face of that polygon then contains surface information.

[0261] Neighborhood

[0262] A region with a preset neighborhood size of n is called an n-ring neighborhood, which defines the area around a vertex based on the connectivity of adjacent faces. Figure 7 As shown, vertex p i The n-ring neighborhood of the vertex p i Starting from the grid edge, all vertices that can be reached within n traversal steps, for example, the 1-ring neighborhood consists of the vertices with the vertex p iThe 2-ring neighborhood consists of directly connected vertices (sharing an edge), and the 2-ring neighborhood consists of the vertices with the vertex p i The number of sides between which the polygon is separated is not greater than 2.

[0263] The parameter n, which controls the size of the n-ring neighborhood, is configurable. Larger values of n expand the neighborhood, capturing a wider range of surface features, while smaller values of n reduce the neighborhood, focusing on local details.

[0264] The adjacency relationship of the n-ring neighborhood can be calculated using the sparse adjacency matrix A. The specific formula is as follows:

[0265]

[0266] The expansion of the neighborhood is achieved by iteratively multiplying the adjacency matrix. The specific formula is as follows:

[0267]

[0268] in, is a vertex in the k-ring neighborhood, A k represents the kth power of matrix A, δ i is the vector representing the initial vertex.

[0269] By changing the parameter n that controls the size of the n-ring neighborhood, the scope of the analysis can be set to adapt to local or global geometry. Smaller ring neighborhoods limit the calculation to a closer neighborhood and retain finer details, while larger ring neighborhoods include a wider surface environment and can capture global structure at the expense of higher computational cost, thus balancing detail preservation and computational efficiency.

[0270] In some examples of this disclosure, the three-dimensional model is a voxel model.

[0271] The voxel model is composed of a set of voxels, and the candidate sampling point is the center of the outermost voxel of the voxel model.

[0272] The neighborhood for determining candidate sampling points may include:

[0273] With the candidate sampling point as the center, the range with a distance no greater than r from the candidate sampling point is determined as the neighborhood of the candidate sampling point.

[0274] The preset neighborhood size value is r, and r is a positive number.

[0275] Specifically, the voxel model simulates the surface geometry of the target object based on voxels, discretizing the 3D object into a structured grid of cubic units (voxels). The resolution of the voxel model is controlled by the voxel_resolution parameter, which determines the number of divisions along the longest axis of the bounding box. Higher resolution can provide finer sampling, but at the cost of increased computational complexity.

[0276] In some examples, a polygonal mesh model can be used to generate a voxel model. For example, the polygonal mesh model can be converted to a voxel model before sampling. Given that voxels in a voxel model are typically very dense, in some embodiments, only a portion of the surface voxels (e.g., the outermost voxels) can be used for further sampling. In some embodiments, when the target object whose surface contour is represented by a sphere is a flexible object (e.g., a human hand), the vertex indices of the polygonal mesh model can be used for sampling, and the conversion from the polygonal mesh model to the voxel model is not performed.

[0277] In some examples, generating a predetermined number of balls may include:

[0278] - Based on the value of the density weight, determining a preset number of sampling points from all candidate sampling points.

[0279] -Generate a preset number of spheres with the determined sampling point as the sphere center.

[0280] For example, determining a preset number of sampling points from all candidate sampling points may include:

[0281] a) Determine one of the candidate sampling points as the initial sampling point.

[0282] b) determining the next sampling point from the remaining candidate sampling points based on the Euclidean distance between each remaining candidate sampling point and the last candidate sampling point determined as the sampling point, and the density weight of each remaining candidate sampling point, until the predetermined number of sampling points is determined.

[0283] The remaining candidate sampling points are candidate sampling points that have not yet been determined as sampling points.

[0284] Density-Weighted Farthest Point Sampling

[0285] In an optional embodiment, step b may include: determining a comprehensive distance between each remaining candidate sampling point and the candidate sampling point last determined as the sampling point based on the Euclidean distance between each remaining candidate sampling point and the candidate sampling point last determined as the sampling point, and a density weight of each remaining candidate sampling point; and determining the remaining candidate sampling point having the largest corresponding comprehensive distance value as the next sampling point.

[0286] For example, the weighted distance can be determined according to the following formula:

[0287]

[0288] Among them, d′(p i ,p i ) Alternative sampling point p i With the last determined sampling point p j The comprehensive distance between i ,p j ) is the alternative sampling point p i With the last determined sampling point p j The Euclidean distance between is the alternative sampling point p i The density weight of .

[0289] j next =max i (d′(p i ,p j ))

[0290] The next sampling point j determined or selected next The maximum comprehensive distance value d′(p i ,p j ) of the sampling point p i .

[0291] The above formula is the formula of the DWFPS (Density-Weighted Farthest Point Sampling) strategy. The DWFPS strategy iteratively selects the vertex farthest from the last determined center of the sphere, and combines the density weight to bias the process of selecting the next center of the sphere towards areas with higher vertex distribution density (that is, areas with more complex geometry). The DWFPS strategy ensures that the sampling frequency is higher in areas with high vertex distribution density, and the sampling frequency is lower in areas with low vertex distribution density. Therefore, this method realizes adaptive sampling, which can effectively capture fine details and large-scale structures without redundant calculations. The DWFPS strategy can ensure uniform spatial coverage while giving priority to areas with higher complexity.

[0292] Unlike the SDF (Signed Distance Function) method, which uses a lot of memory because it generates a surface distance field function that fits any voxel in the 3D space of interest, the sphere-based calculations in this disclosure are performed pairwise between only a small number of spheres represented as a 3D point cloud. Consequently, the disclosed method uses very little CPU / GPU memory, yet is sufficient to generate cost / reward functions that can significantly alter the robot's motion.

[0293] Adaptive sphere radius

[0294] The size of the balls can be adjusted accordingly based on changes in the surface contour of the target object. The method further includes: determining the radii of a preset number of balls based on the three-dimensional model, wherein the radii of the balls in different regions correspond to the geometric complexity of the surface contour of the different regions.

[0295] Optionally, the spherical radius of the balls in different regions corresponding to the geometric complexity of the surface profiles of the different regions includes: the spherical radius of the balls in the first region is smaller than the spherical radius of the balls in the second region. The sampling points may include a first sampling point and a second sampling point.

[0296] Optionally, determining the sphere radius of the preset number of spheres may include: determining the radius of each sphere based on the preset radius and a density weight of a sampling point corresponding to each sphere. If the density weight of a first sampling point is greater than the density weight of a second sampling point, the radius of the sphere corresponding to the first sampling point is smaller than the radius of the sphere corresponding to the second sampling point.

[0297] Optionally, the density weight value of the sampling point corresponding to each sphere may be normalized to obtain a normalized density weight value of the sampling point corresponding to each sphere, wherein the normalized density weight value is a positive number less than 1.

[0298] Specifically, to ensure that the value of the density weight remains bounded, the density weight can be normalized using the following formula:

[0299]

[0300] Among them, w i is the density weight, is the normalized density weight.

[0301] Through the above normalization process, the value of the density weight can be mapped to a range between 0 and 1, thereby preventing extreme changes in the value of the density weight and ensuring the stability of the value of the density weight.

[0302] Optionally, in order to normalize the density weight values of the sampling points corresponding to each sphere, the method may further include: determining the spherical radius value of each sphere according to the preset radius value and the normalized density weight values of the sampling points corresponding to each sphere.

[0303] For example, the value of the spherical radius of each sphere is determined according to the value of the preset radius, the value of the normalized density weight of the sampling point corresponding to each sphere, and the value of the proportional coefficient.

[0304] The proportional coefficient is used to control the sensitivity of the value of the sphere radius of each sphere to the change of the value of the normalized density weight of the sampling point corresponding to each sphere.

[0305] Specifically, the value of the sphere radius can be determined according to the following formula:

[0306]

[0307] Among them, r0 is the preset radius, which can be defined by the user. is the normalized density weight, and s is the proportional coefficient.

[0308] The scaling factor s is used to control the sensitivity of the sphere radius to changes in the value of the density weight. x Ensure that the sphere radius r i The scale factor s can change smoothly and continuously without sudden jumps, allowing the sphere to seamlessly adapt to local geometric changes. In some embodiments, the scale factor s can be set to 10 and the result can be visually inspected by overlaying the generated sphere with the polygonal mesh model of the target object. If the sphere is too small, the scale factor s can be increased. If the sphere is too large, the scale factor s can be decreased. In some embodiments, the scale factor s can be 0.1, 1, 10, 100, 1000, or any value between 0.1 and 1000.

[0309] Through the above formula, the sphere used to represent the surface contour of different areas of the target object can shrink in areas where the candidate sampling points are densely distributed, and expand in areas where the candidate sampling points are sparsely distributed, thereby achieving adaptive coverage without introducing unnecessary computational costs.

[0310] Voxel-based sampling

[0311] In some embodiments, the 3D model may be, for example, a voxel model. In cases where the 3D model is a voxel model, the density weights may be normalized according to the above formula, and the sphere radius may be determined based on the normalized density weights and the above formula. Because the sphere radius scales with the local voxel density, it allows for finer detail in high-density areas and larger spheres in sparse areas. In cases where uniformity is prioritized over adaptability, a fixed sphere radius may be used instead of determining the sphere radius using the above method.

[0312] Using a voxel model as a 3D model has the following advantages over using a polygonal mesh model as a 3D model:

[0313] 1. Works independently of the mesh topology, allowing the ball to be placed outside the mesh to capture volume features and discontinuous areas.

[0314] 2. Provides uniform coverage, making it effective for sparse or low-resolution meshes.

[0315] Optionally, descriptor information of a preset number of spheres may be generated. The descriptor information includes at least one of index information, position information, normal information, and sphere radius. The index information is an index list, which is an index list of candidate sampling points {1, 2, 3, ..., n IH} subset, n IH is the total number of balls representing the surface contour of the target object. These descriptor information can be seamlessly integrated with tasks such as collision detection.

[0316] Optionally, the method of generating a preset number of balls to represent the surface contours of different regions of the target object further includes: 1) acquiring the position information of all candidate sampling points in real time, 2) determining the position information of all sampling points based on the position information and index information of all candidate sampling points. For example, once a specific set of hand mesh vertex indexes I is selected, H (i.e. the complete hand mesh vertex index {1,2,3,…,n IH}Integer index i within the subset H List, n IH The total number of hand mesh vertices) is used as the mounting point of the hand surface sphere (this set of indexes only needs to be generated once and can be reused continuously), and the position of the hand surface sphere can be continuously calculated in real time. The complete hand mesh vertices generated by the simpleHand model in the world coordinate system are represented as V hand , the center position of the wrist surface sphere in the world coordinate system is expressed as S hand , then S can be easily computed at any time using array index selection operations hand =V hand [I H ].

[0317] Generate surface spheres to represent flexible objects

[0318] The surface sphere representations generated for robotic components, rigid objects, and flexible objects (such as a human hand) in a world coordinate system can be directly used for different types of real-time robotic control operations. For flexible objects (such as a human hand), the following steps can be used to obtain a 3D model of the target object.

[0319] a. Obtaining real-time video stream data. For example, a real-time video stream is obtained by capturing a camera. In the video stream, there is a two-dimensional image of the target object. Thus, a two-dimensional image of the target object can be captured in real time.

[0320] b. Generate a three-dimensional model of the target object based on the two-dimensional image of the target object.

[0321] Those skilled in the art will appreciate that there are different ways to generate a three-dimensional model of a target object based on a two-dimensional image of the target object. For example, in some embodiments, the following method steps may be used to generate a three-dimensional model.

[0322] c. Determine a bounding box for the target object in the two-dimensional image. The bounding box for the target object in the two-dimensional image can be determined based on an open vocabulary object detection model and input information corresponding to the target object. For example, the two-dimensional image can be input into an open vocabulary object detection model (e.g., "hand"). Using the vocabulary corresponding to the target object, the target object can be identified in the two-dimensional image, and the position and size of the target object in the two-dimensional image can be determined. Based on the position and size, a bounding box is generated in the two-dimensional image to enclose the target object within the bounding box. The bounding box can indicate the position and size of the target object in the two-dimensional image.

[0323] d. Generate a 3D model of the target object based on the 2D image and the bounding box of the target object.

[0324] Taking the target object as a hand as an example, step d may include:

[0325] 1) Determine the 2D hand joints and 3D hand vertex positions of the target object.

[0326] 2) Obtain a three-dimensional model based on the 2D hand joints and 3D hand vertex positions.

[0327] like Figure 8 An example of constructing a three-dimensional model of a hand from a camera image in real time is shown, wherein the three-dimensional model may be, for example, a polygonal mesh model.

[0328] Step 710: Input an image. For example, an RGB image captured by a camera can be passed to an open-vocabulary object detection model, such as YOLO-World or GroundingDINO. The open-vocabulary object detection model YOLO-World is described in detail in the document “Cheng, Tianheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. "Yolo-world: Real-time open-vocabulary object detection." In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 16901-16911.2024,” the entire disclosure of which is incorporated into this disclosure. The document “Ren, Tianhe, Qing Jiang, Shilong Liu, Zhaoyang Zeng, Wenlong Liu, Han Gao, Hongjie Huang et al.”Grounding DINO 1.5: Advance the “Edge” of Open-Set Object Detection.” arXiv preprint arXiv:2405.10300(2024)” describes the open-set object detection model GroundingDINO in detail, and all the disclosed contents are incorporated into the present disclosure.

[0329] Step 720: Object detection. For example, an open vocabulary object detection model can be used to detect the possible location and size of a hand, i.e., a bounding box of a hand. Alternatively, the open vocabulary object detection model can use the keyword "hand" as input to locate and detect hands in the input image. These open vocabulary object detection models can detect a variety of object categories, including hands, without the need for retraining.

[0330] Step 730: Joint and vertex estimation. For example, the bounding box labeled image can be processed by the simpleHand model (consisting of a label generator and a mesh regressor) to estimate the 2D hand joints and 3D hand vertex positions, respectively. The simpleHand model is described in detail in the paper “Zhishan Zhou, Shihao Zhou, Zhi Lv, Minqiang Zou, Yao Tang, and Jiajun Liang. A simple baseline for efficient hand mesh reconstruction. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1367–1376, June 2024,” the entire disclosure of which is incorporated into this disclosure. Furthermore, the local camera coordinates of the hand vertices can be converted to 3D world coordinates together with the depth information.

[0331] Step 740: Mesh reconstruction. For example, a hand mesh can be created based on the hand vertices obtained in step 730 and a triangle mesh generated using a hand model (e.g., a MANO model). The MANO model is described in detail in the document "Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: modeling and capturing hands and bodies together. ACM Transactions on Graphics, pages 1–17, 2017," the entire disclosure of which is incorporated herein.

[0332] The human hand is a highly flexible and deformable structure with multiple joints and degrees of freedom. Unlike rigid objects, its geometry is constantly changing due to the bending, stretching, and rotational movements of the fingers and palm. Therefore, it is impractical to hard-code the position of the sphere used to represent the surface contour of the hand in a stationary coordinate system, as it cannot accurately capture the deformation associated with the posture. To address this problem, the adaptive surface sphere representation mentioned above can be utilized to dynamically determine the vertex indices of the spheres used to represent the surface contour of the hand on the simpleHand mesh template. This approach ensures that the spheres used to represent the surface contour of the hand adapt to local geometric changes and deformations in real time.

[0333] Anti-collision loss function

[0334] The inventors further discovered that to efficiently calculate potential collision risks between the robot and objects in its environment, they can use the principles described above to represent the robot's linkages and object geometry as a set of small spheres, each defined by its center and radius. This approach breaks down complex shapes into simpler components, making distance calculations and collision checks straightforward.

[0335] Specifically, each link of the robot and each object in the environment is approximated by a set of spheres. Let The set of spheres that approximate the robot geometry, where is the center of the i-th sphere associated with the robot, is the radius of the i-th sphere. Similarly, let A collection of spheres representing each object, where is the center of the jth sphere associated with the robot, r j obj is the radius of the i-th sphere.

[0336] By using these sphere representations, we can simplify the process of determining potential collisions between robots and objects. The positions of these spheres are adjusted in real time based on the robot's motion, and by calculating the distances between all robot-object sphere pairs, we can continuously and efficiently check for collisions.

[0337] Figure 9 Some examples of collision sphere representations are shown, with the original mesh of the object shown above and the small sphere 510, 520 and 530 representations of the corresponding objects shown below.

[0338] Robot-object collision loss function

[0339] Using small sphere representations of the robot and object, along with their respective poses, a robot-object collision loss can be computed, enabling efficient collision avoidance. This approach simplifies the complex geometry of the robot links and objects into manageable spherical regions, allowing for efficient computation of pairwise distances between all robot-object sphere combinations.

[0340] To deal with the collision avoidance problem efficiently, we first represent the geometry of the robot links and the object as a set of small spheres in each local coordinate system, defined by their center positions and radii. He Ran Represent the collision sphere sets of the robot and the object respectively. and is the center of the i-th robot sphere and the j-th object sphere, and Denotes the corresponding radius. N is the number of robot links, M is the number of objects in the scene, and B is the number of balls per link or object.

[0341] Since the poses of the robot and the object change dynamically, the center of each sphere must be transformed from its local coordinate system to the robot reference (world) coordinate system to reflect the current configuration of the robot and the object. This transformation is achieved using the robot's forward kinematics and the pose matrix of each object. Let FK n (θ′) represents the forward kinematic transformation matrix of the nth robot link under a given joint configuration θ′, and is the 6D pose transformation matrix of the target object from the object's rest coordinate system to the robot's rest coordinate system, which can transform the center of the sphere:

[0342]

[0343] After transforming all sphere centers to the global coordinate system, calculate the pairwise effective distance D between each robot-object sphere pair (i, j) ij , as shown below:

[0344]

[0345] This distance D ij Represents the distance between the surfaces of two spheres.

[0346] In an optional embodiment, the robot-object collision loss L robot-obj As a sub-loss function in the loss function.

[0347] Specifically, to define a smooth collision cost that penalizes close proximity and overlap, a cosine-based penalty function f can be used cos (D ij ), when D ij When the threshold distance skin_dist is exceeded, the function decays smoothly to zero. The penalty function is defined as:

[0348]

[0349] Robot-object collision loss L robo_obj Computed as the average of the individual collision costs across all robot-object sphere pairs:

[0350]

[0351] This loss function represents the overall collision risk in the workspace. By trajectory optimization, L robot-obj It effectively encourages robots to avoid potential collision areas, thereby promoting safe and efficient movement in environments with multiple obstacles.

[0352] Robot-human collision loss function

[0353] The human body is a flexible object. As mentioned above, combined with human pose estimation algorithms, the human body can also be represented as a set of spheres, similar to how objects are modeled. These spheres can then be transformed based on the estimated human pose to accurately reflect the body's configuration in space. Using this representation, a collision loss function can be calculated for robots with flexible objects such as the human body, similar to the collision loss function for objects. This enables the system to account for potential interactions or collisions with humans, thereby enhancing safety and adaptability in dynamic environments.

[0354] Robot self-collision loss function

[0355] In an optional embodiment, the robot's self-collision loss L self-collision As a sub-loss function in the loss function.

[0356] Specifically, to calculate the robot's self-collision loss, we can use a similar approach to the robot-object collision loss, but only consider the robot's internal structure. Each link of the robot is represented by a set of small spheres, each of which has a center position and radius. Let represents the set of collision spheres across all links. Since the robot configuration changes dynamically, the forward kinematic transformation matrix FK for each link i is used n (θ′) transforms the center of each sphere from its local link coordinate system to the global (robot reference) coordinate system. The transformed center of each sphere is then given by given.

[0357] After transforming all sphere centers into the global coordinate system, calculate the effective distance D between each pair of (i, j) spheres belonging to different links ij ,Right now In order to obtain a smooth self-collision cost, the same cosine-based penalty function f can be adopted cos (D ij ), when D ij Beyond the threshold distance skin_dist, the function decays smoothly to zero.

[0358] Finally, the total self-collision loss L self-collision Computed as the average of all pairwise self-collision costs over all links, excluding pairs within the same link:

[0359]

[0360] Minimize L by trajectory optimization self-collision The robot can be encouraged to adopt joint configurations that avoid self-collision, promoting safer motion and internal coordination across links.

[0361] Loss function for collision between robot and physical surface

[0362] In an optional embodiment, the robot can collide with the physical surface loss function L robot-surface As a sub-loss function in the loss function.

[0363] Specifically, in addition to avoiding collisions between the robot and objects and avoiding collisions with itself, the robot also needs to avoid collisions with some physical surfaces in the environment, such as tables, floors, or walls. These surfaces can be represented as a set of planes in 3D space, each defined by a normal vector and an offset distance from the origin. Let each physical surface S k By its normal vector and offset d k Expressed as, so the plane equation of each surface is in is any point on the plane.

[0364] To calculate the collision risk between the robot and the surface, the shortest distance between each collision ball on the robot and the surface in the environment is measured. in is the center of the collision sphere in the global coordinate system, D ik To surface S k The distance is calculated as follows:

[0365]

[0366] This effective distance D ik Represents the sphere surface and the physical surface S k The space between them; in order to obtain a smooth self-collision cost, the same cosine-based penalty function f is used cos (D ik ), when D ik Beyond the threshold distance skin_dis, the function decays smoothly to zero.

[0367] Then, the total collision loss L between the robot and the physical surface is robot-surface Calculated as the average of these individual collision costs over all robot spheres and associated physics surfaces:

[0368]

[0369] In some examples of this disclosure, the loss function Loss can be shown as follows:

[0370] Loss=λλ1L dist +λ2L dist +λ3L n_dor +λ4L robot-obj +λ5L self-collision

[0371] +λ6L robot-surface +λ7L angle-limit +λ8L speed-limit

[0372] Each term in the loss function penalizes deviations from the desired pose, orientation, location, and spatial constraints, including collision risks with objects, surfaces, and the robot itself, as well as joint restrictions. In some examples of this disclosure, if the object is a flexible object such as a human body, the loss function Loss may further include the sub-collision loss functions for the robot and the flexible object, such as the human body, and their corresponding weights, as described above.

[0373] Using this loss function, the joint angles θ are updated via gradient descent to generate the robot trajectory. This process continues to produce updated θ until each loss term is below a predefined tolerance, indicating that the robot’s current pose closely matches the desired grasp pose.

[0374] In some examples disclosed herein, when the value of each sub-loss function in a loss function falls below a predetermined threshold, the robot may receive a first instruction instructing the end effector to perform an object-grasping operation. For example, the first instruction may be a CLOSE() command, which is sent to the gripper to close and grasp the object. This method provides a robust and effective approach for achieving collision-free, precise grasping in dynamic environments.

[0375] The robot can also receive other instructions. For example, when the robot's task is to grasp the target object, a second instruction can be sent to the end effector. For example, the second instruction is a CLOSE() command to close the end effector and grasp the target object. This method provides a robust and efficient way to accurately grasp the target object in a dynamic environment. For example, the third instruction can be a GOTO() command, and the fourth instruction can be an OPEN() command. The GOTO() command combined with the CLOSE() command can achieve efficient and automatic object grasping. For automatic placement, the GOTO() command can also be used to move the end effector to the target pose, and then the OPEN() command can be used to release the object.

[0376] Figure 10 The figure shows the process of the gripper at the end of the robot grasping the target object. The white ghost is the digital robot and object rendered in real time using the pose estimation results. Figure 10 (a) to Figure 10 (d), The real robot’s gripper gradually approaches the target grasping pose. Figure 10 (e) shows the gripper executing the close command and grasping the teapot. Figure 10 (f) shows the robotic arm lifting the teapot, indicating a successful grasp.

[0377] like Figure 11 As shown, in some examples of this disclosure, determining the loss function S130 also includes:

[0378] S820: Obtain state information of the robotic arm. The state information may indicate one or more of the following: whether the robotic arm is holding an object, whether the robotic arm is moving, whether the robotic arm is stationary, whether the current posture of the robotic arm matches the target posture, whether the robotic arm is grasping an object, or whether the robotic arm is releasing the held object.

[0379] S830. Adjust the weight value corresponding to each sub-loss function of the loss function according to the state information of the robotic arm.

[0380] By adjusting the weights associated with each sub-loss function based on the robot's state, the number and type of sub-loss functions in the loss function can be adjusted. For example, the number and type of sub-loss functions can be adjusted by adjusting the corresponding weights based on the type of control command and the presence of a load on the end effector.

[0381] Optionally, in some examples of this disclosure, determining the loss function S130 may include:

[0382] S810: According to the instruction received by the robotic arm and whether the end effector grasps the object, the state information of the robotic arm can be determined.

[0383] In some examples disclosed herein, the robot's state information can be determined based on the type of command received, the proximity to the target object, and whether the end effector is holding a payload. This is important for optimizing control and safety, as well as effectively managing the objectives defined by the loss function. The robot's state affects the priority of different constraints and losses in the objective function, that is, it affects the weight coefficients corresponding to the various sub-objective functions in the objective function, allowing the robot to adjust its behavior according to the requirements of the current task. The criteria for determining the robot's state include whether the end effector is moving while holding a payload and whether the end effector is grasping or releasing an object. Depending on the type of low-level command the robot receives and whether the end-effector is holding a payload, the robot can be defined into eight states: IDLE_WITHOUT_PAYLOAD (stationary without payload), IDLE_WITH_PAYLOAD (stationary with payload), MOVING_WITH_PAYLOAD (moving with payload), MOVING_WITHOUT_PAYLOAD (moving without payload), PRE_GRASPING (pre-grasping), GRASPING (grasping), PRE_RELEASING (pre-releasing), and RELEASING (releasing). These eight states are managed through a state machine framework that dynamically transitions between them based on the type of command received and whether the end-effector is holding a payload. This state machine architecture allows system transitions to ensure optimized task-specific operations based on the robot's current role and environment.

[0384] Figure 12 The figure shows the transition relationship between the eight states mentioned above. IDLE_WITHOUT_PAYLOAD and IDLE_WITH_PAYLOAD both indicate that the robot is in idle mode, waiting for the next command. The difference between IDLE_WITHOUT_PAYLOAD and IDLE_WITH_PAYLOAD is that IDLE_WITHOUT_PAYLOAD indicates that there is no payload in the end effector, while IDLE_WITH_PAYLOAD indicates that the end effector is holding a payload. After the robot completes a task, the default robot state is IDLE_WITHOUT_PAYLOAD or IDLE_WITH_PAYLOAD. If the end effector holds a payload after successfully completing a grasping operation, the robot enters IDLE_WITH_PAYLOAD; if the end effector does not hold a payload, it enters IDLE_WITHOUT_PAYLOAD.

[0385] When a robot receives a GOTO() command, the robot's state transitions to either MOVING_WITHOUT_PAYLOAD or MOVING_WITH_PAYLOAD. If the end-effector is holding a payload, the robot transitions to MOVING_WITH_PAYLOAD; if the end-effector is not holding a payload, the robot transitions to MOVING_WITH_PAYLOAD. In MOVING_WITH_PAYLOAD, additional collision constraints are activated to account for the payload's interactions with surrounding objects and surfaces, protecting both the robot and the payload. In contrast, in MOVING_WITHOUT_PAYLOAD, the system reduces the emphasis on collisions with external constraints and instead focuses on efficient movement to the next operational position.

[0386] When the end effector is required to grasp or release an object, the state machine changes the robot's state from the pre-action state to the execution state; specifically, when the end effector is about to grasp an object, the state machine changes the robot's state from PRE_GRASPING to GRASPING; when the end effector is about to release an object, the state machine changes the robot's state from PRE_RELEASING to RELEASING, thereby achieving accurate and safe grasping of the object.

[0387] When the robot is in PRE_GRASPING, it prepares to align the end-effector and accurately approach the target object, optimizing its positioning before grasping. After the end-effector reaches the target position and receives the CLOSE() command, the robot transitions to GRASPING, where it can safely grasp the object. Similarly, when the robot is in PRE_RELEASING, it prepares to accurately position itself for placement in the designated area. When the robot prepares to release the object held by the end-effector, upon receiving the OPEN() command, the state machine transitions the robot's state to RELEASEING, enabling accurate and safe placement of the object. When the robot is in PRE_GRASPING or PRE_RELEASING, the type, number, and weight coefficients of the sub-objective functions in the objective function are adjusted accordingly to minimize pose error.

[0388] By adjusting the type, number, and weight coefficients of the sub-objective functions in the objective function based on the robot's state, a smooth, safe, and adaptable trajectory can be obtained for the end-effector, thereby supporting the efficient execution of complex manipulation tasks. This state-based configuration enables the robot to maintain a high level of reliability and adaptability in dynamic environments, because each state is paired with a set of optimization constraints that meet the operational requirements of the current task.

[0389] Real-time pose estimation is crucial for enabling precise and adaptive robotic manipulation, especially in dynamic environments where objects may move or change their spatial relationships relative to each other or the robot. For tasks such as pick-and-place, assembly, and bundling, it is crucial for the robot to accurately estimate the pose of individual objects and the interactions between objects and the robot. This becomes even more important when managing multiple objects or handling complex manipulations, such as attaching or detaching objects to or from the robot's end-effector.

[0390] Figure 13 The interaction process between the robot and the object during the grasp-and-place operation is shown. The white ghost is the digital robot and object rendered in real time using the pose estimation results. Figure 10 (a) to Figure 10 (b) The gripper at the end of the actual robot gradually approaches the handle of the teacup. Figure 10 In (c), the gripper successfully grasps the teacup, attaching the object to the gripper. Figure 10 (d) to Figure 10 (e), the position of the teacup is continuously updated according to its relative position with the gripper. Figure 10 In (f), the grippers release the cup, separating it from the grippers.

[0391] Object pose estimation

[0392] Object pose estimation is the process of determining the position and orientation of an object in space, that is, the process of determining the object's 6D pose, usually from image data, by estimating the pose of the target object in a given image relative to the camera that captured the image. Object pose estimation is a key technology in fields such as augmented reality, virtual reality, robotic manipulation, and hand-object interaction. Depending on the application requirements, the pose of the target object is estimated to different degrees of freedom (DoF), such as 3DoF, which only includes 3D rotation, 6DoF, which also includes 3D translation, or 9DoF, which includes estimating the object's 3D size in addition to 3D rotation and 3D translation. Understanding the spatial arrangement of objects in the environment can help robots perform operations on the target object.

[0393] A real-time pose estimation system can provide direct input to a robot's control system, enabling it to perform precise movements. For example, during object grasping, pose data enables the control system to calculate optimal approaches and maneuvering strategies to minimize the risk of collisions or slippage. Pose estimation can also go beyond object recognition and visual tracking alone to incorporate feedback from the robot's own sensors. Signals such as the timing of attachment and detachment events, payload status, and joint positions are crucial for updating the object's pose relative to the robot's end-effector. This integration can significantly improve pose estimation accuracy. This is because the object's pose is no longer estimated solely through external observation but is directly linked to the robot's precise motion data, reducing errors caused by occlusions or visual noise. Furthermore, when an object is attached to the robot, the system can utilize more advanced manipulation strategies, such as treating the combined object-robot structure as a single entity for collision avoidance and path planning. Therefore, in some examples of this disclosure, after the end-effector grasps an object, a sphere representing the end-effector's surface contour is stacked with a sphere representing the object. The stacked spheres represent the object as an extension of the end-effector, representing the surface contour of the extended end-effector.

[0394] In more complex application scenarios, such as when the robot needs to complete the task of binding multiple objects, the robot needs to maintain the relative stability of each object while managing the collective pose of the group of objects. Real-time feedback data from sensors mounted on the robotic arm enables the robot to detect and compensate for unexpected events such as slippage, deformation, and external interactions. The real-time feedback data can include tactile data and proprioception data. By combining visual pose tracking and feedback data from sensors mounted on the robotic arm, the robot's control strategy can be dynamically adjusted to ensure the robustness and precision of the interaction between the object and the robot. The integration of this multimodal data is crucial to improving the overall accuracy of manipulation tasks in complex and unpredictable environments.

[0395] In an optional embodiment, the planning method further includes: receiving image information of all objects and the robotic arm in the working space of the robotic arm sent by the image acquisition module; and displaying the image information on a user interface of the display.

[0396] In an optional embodiment, the planning method further includes generating instructions, such as control commands, in response to user operations on the user interface. For example, the user interface may be a GUI (Graphical User Interface), where the user can conveniently and intuitively perform operations on the GUI.

[0397] Graphical User Interface (GUI)

[0398] The graphical user interface (GUI) allows users to easily and intuitively control the robot. The GUI displays video from each camera and a rendering of the 3D scene. To control the robot, users simply select commands from the toolbar and click on the interactive video panel.

[0399] Figure 14 A graphical user interface 910 is shown for users to easily and intuitively control the robot. The graphical user interface displays the video from each camera and a rendering of the three-dimensional scene. To control the robot, the user simply selects commands from the graphical user interface's toolbar and clicks on the interactive video panel.

[0400] The GUI system is designed for intuitive and user-friendly operation, enabling seamless interaction with a real-time digital twin reconstruction of the robot and its environment. Users can issue high-level task commands directly in the GUI by visually interacting with the reconstructed scene. For example, users can left-click an object to select it, move it to a new location, and optionally adjust its rotation before placing it. After entering the desired command, the user clicks the "Robot Start" button to execute the command. Users can enter multiple high-level task commands before starting. On the backend, a processor converts these high-level commands into a series of low-level commands, such as GOTO(), OPEN(), and CLOSE(), which serve as input to the trajectory planning method of this embodiment for the robot to execute. The robot, whether physical or simulated, executes these commands, and its movements are dynamically reflected in the real-time digital twin reconstruction of the GUI. The GUI also records an event log, providing users with feedback on the success or failure of each high-level task, ensuring transparency and facilitating troubleshooting.

[0401] In an optional embodiment, the planning method further includes: receiving image information of all objects and the robotic arm in the working space of the robotic arm sent by the image acquisition module; rendering the image information to obtain rendering information; sending the rendering information to the display for real-time display; and generating instructions in response to user operations on the user interface.

[0402] The inventors found that the initial implementation of the current GUI visualization pipeline was inefficient, severely impacting the frames per second (FPS) of the entire system, including tasks such as mask segmentation, pose estimation, and control processing. The main bottleneck was inefficient data transfer between the CPU and GPU, and frequent memory copying led to latency spikes and performance degradation. The inventors also found that the rendering pipeline ran in synchronous blocking mode, preventing other processing threads from executing in parallel and causing further delays. Using an unoptimized GUI framework exacerbates these problems because it fails to take advantage of modern GPU acceleration technologies (such as direct GPU memory sharing), resulting in excessive CPU utilization and slower rendering times for scene meshes and surface spheres.

[0403] Using the spheres generated in some examples of this disclosure to represent the surface contours of the target object can optimize the rendering pipeline. For example, memory sharing can be achieved by utilizing the PyTorch-OpenGL integrated GPU. This method implements zero-copy data sharing between PyTorch tensors and OpenGL buffers, eliminating redundant memory transfers and enabling direct GPU rendering of visual output. The solutions provided in some embodiments of this disclosure can implement custom OpenGL shaders, thereby enabling transformation and lighting effects to be processed directly on the GPU, reducing the computational burden on the CPU. The solutions provided in some embodiments of this disclosure can utilize parallel threads to separate rendering tasks from data processing, ensuring that GUI updates no longer hinder critical system operations. In addition, the frame rate cap mechanism dynamically adjusts GUI updates based on system load to maintain FPS stability. Further optimizations include lazy loading and caching strategies for static meshes to reduce redundant calculations and memory usage. Together, these enhancements create a scalable, responsive, and real-time visualization system that can run seamlessly with other computationally intensive tasks.

[0404] Example 2

[0405] This embodiment provides a control system 1000 for a robotic arm, such as Figure 15 As shown, the control system includes: one or more processors 1020, and the one or more processors are used to execute computer programs to control the robotic arm to implement the control method in Example 1.

[0406] In some examples of the present disclosure, the control system 1000 may further include an image acquisition module 1010. The image acquisition module is used to acquire image information of all objects and the robotic arm within the robotic arm's workspace. The image acquisition module 1010 may send the image information to the processor 1020 for processing.

[0407] For example, the image acquisition module 1010 may be implemented using a camera. The image acquisition module 1010 may also include a depth camera.

[0408] In some examples of the present disclosure, the control system 1000 may further include: a robotic arm 1030 and a sensor 1040. The robotic arm includes a base, at least one link, and a joint connecting the at least one link, wherein at least one link includes an end effector. The sensor 1040 may be disposed on the end effector of the robotic arm, for example, a tactile sensor.

[0409] The processor 1020 can also be used to: receive image information of all objects and the robotic arm in the workspace of the robotic arm 1030 sent by the image acquisition module 1010; obtain feedback information sent by the sensor 1040; and determine whether the end effector of the robotic arm 1030 holds an object based on the image information and feedback information.

[0410] In some examples of the present disclosure, the control system 1000 may further include a display 1050, which is used to display a user interface and to display in real time within the user interface the image information processed by the processor 1020. The user interface may also be in the form of a GUI.

[0411] In some examples of the present disclosure, the display 1050 may also be used to: receive user operations through a user interface; and the processor 1020 may also be used to generate control commands according to the user operations.

[0412] In an optional embodiment, the image acquisition module and / or the display: The image acquisition module is used to obtain image information of all objects and the robotic arm in the workspace; the display is used to display the user interface and display the received three-dimensional rendering information in real time in the user interface.

[0413] In some embodiments, through multithreading, the system can include a robust architecture to manage real-time operations and interactions to ensure seamless coordination between robot control, user input, and real-time digital twin visualization.

[0414] Figure 16 The multi-threaded system structure in some embodiments is shown. Each solid line represents a queue for a thread or process to send information.

[0415] The GUI thread serves as the main interface, receiving real-time SceneInfo from the Pose thread to display 3D models of the robot, objects, and its environment. It also handles all user interactions, allowing the user to generate high-level commands, which are then sent to the Plan thread for processing.

[0416] In the robot communication process, the robot API continuously queries the robot hardware for its current θ and sends the desired θ (actual-q) calculated by the control thread.

[0417] This process also monitors the state of the robot arm end effector, sending “close” or “open” signals as needed to perform a grasping or releasing operation.

[0418] The planning thread is responsible for breaking down high-level user commands into a series of low-level commands and combining them with waypoint-based robotic arm movements for precise execution. For each low-level command, the planning thread dynamically creates a corresponding control thread and passes the robot's state to it for execution. The control thread uses high-level optimization techniques to calculate the desired robot trajectory and execute the low-level tasks. After each execution, the control thread returns a result indicating the success or failure of the move, which the planning thread uses to continue planning, update the robot's state, and log events.

[0419] The pose thread processes camera frames in real time to estimate the pose of the robot and target objects from 2D images. It also classifies and recognizes objects, packaging this data into SceneInfo for distribution to the GUI thread (GUI / mainthread) for real-time digital twin visualization and to the planning and control threads for intelligent robot operations.

[0420] This multi-threaded architecture utilizes asynchronous, non-blocking queues for inter-thread communication, ensuring efficient and real-time data exchange. Threads are dynamically created and destroyed as needed, particularly for tasks that require the execution of low-level commands, optimizing resource utilization and enhancing system responsiveness. This design enables the system to deliver real-time performance, robust task execution, and intuitive user interaction for dynamic and intelligent robotic operations.

[0421] Since the system embodiments generally correspond to the method embodiments, reference will be made to the description of the method embodiments for relevant details. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components of the units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the disclosed solution.

[0422] Example 3

[0423] Figure 17 This is a structural diagram of an electronic device showing an example embodiment of the present disclosure. The electronic device includes a memory, a processor, and a computer program stored in the memory and used to run on the processor. When the processor executes the computer program, the control method of the robotic arm of any of the above embodiments is implemented. Figure 14 The electronic device 30 shown is only an example. For example, those skilled in the art will appreciate that part of the processor, such as an ASIC processor, may include a memory. Figure 14 The electronic device 30 shown should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0424] like Figure 17 As shown, the electronic device 30 may be a general-purpose computing device, such as a server device. Components of the electronic device 30 may include, but are not limited to, the at least one processor 31, the at least one memory 32, and a bus 33 connecting various system components (including the memory 52 and the processor 31).

[0425] The bus 33 includes a data bus, an address bus, and a control bus.

[0426] The memory 32 may include a volatile memory, such as a random access memory (RAM) 321 and / or a cache memory 322 , and may further include a read-only memory (ROM) 323 .

[0427] The memory 32 may also include a program tool 325 (or utility) having a set (at least one) of program modules 324, such program modules 324 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include an implementation of a network environment.

[0428] The processor 31 executes various functional applications and data processing by running the computer program stored in the memory 32, such as the control method of the robotic arm provided in any of the above embodiments.

[0429] The electronic device 30 can also communicate with one or more external devices 34 (e.g., a keyboard, pointing device, etc.). Such communication can occur via an input / output (I / O) interface 35. Furthermore, the electronic device 30 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 36. As shown, the network adapter 36 communicates with other modules of the electronic device 30 via a bus 33. It should be understood that, although not shown, other hardware and / or software modules can be used in conjunction with the electronic device 30, including but not limited to microcode, device drivers, redundant processors, external disk drive arrays, RAID (RAID) systems, tape drives, and data backup storage systems.

[0430] It should be noted that although several units / modules or sub-units / modules of the electronic device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0431] Example 4

[0432] Some embodiments of the present disclosure also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the control method of the robotic arm provided by any of the above embodiments.

[0433] Example 5

[0434] Some embodiments of the present disclosure also provide a computer program product, including a computer program, which implements the above-mentioned control method of the robotic arm when executed by a processor.

[0435] In some embodiments of the present disclosure, the readable storage medium and the computer program product may more specifically include but are not limited to: a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0436] Example 6

[0437] Some embodiments of the present disclosure also provide a computer program, which, when executed by one or more processors, can implement the above-mentioned method for controlling the robotic arm.

[0438] The program code for executing the computer program product of the present disclosure may be written in any combination of one or more programming languages, and the program code may be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on the remote device.

[0439] Although specific embodiments of the present invention have been described above, those skilled in the art will appreciate that these are merely illustrative and that the scope of the present invention is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, and such changes and modifications are intended to fall within the scope of the present invention.

Claims

1. A method for controlling a robotic arm, the robotic arm comprising a base, at least one connecting rod, and a joint connecting the at least one connecting rod, wherein: The at least one connecting rod includes an end effector, and the control method includes: Obtaining the target pose of the end effector in world coordinates and the forward kinematics information of the robotic arm; determining a second current pose of the end effector in world coordinates according to the first joint angle of the robotic arm, the forward kinematics information, and the first current pose of the end effector; determining a loss function, the loss function including a first loss function corresponding to a difference between the target pose and the second current pose; Based on the loss function, the first joint angle is updated by a gradient descent algorithm to control the movement of the robotic arm.

2. The method for controlling a robotic arm according to claim 1, wherein: The target posture includes a first three-dimensional grasping point coordinate, a first grasping direction vector and a first grasping direction normal vector; The second current posture includes a second three-dimensional grasping point coordinate, a second grasping direction vector, and a second grasping direction normal vector.

3. The control method of the robot arm according to claim 2, characterized in that: The first loss function includes: A first sub-loss function, wherein the first sub-loss function corresponds to a difference between the first three-dimensional grasping point coordinates and the second three-dimensional grasping point coordinates, and the first sub-loss function corresponds to a first weight value; a second sub-loss function, the second sub-loss function corresponding to a difference between the first grasping direction vector and the second grasping direction vector, the second sub-loss function corresponding to a second weight value; A third sub-loss function, the third sub-loss function corresponding to the difference between the first grasping direction normal vector and the second grasping direction normal vector, the third sub-loss function corresponding to a third weight value.

4. The method for controlling a robotic arm according to any one of claims 1 to 3, wherein: The control method includes: Determining an updated second current pose of the end effector in world coordinates according to the updated first joint angle, the forward kinematics information, and the first current pose; Obtaining a first velocity of the end effector according to the second current pose, the updated second current pose, and a time step of the gradient descent algorithm; determining a predicted pose of the end effector in world coordinates based on the first velocity of the end effector, the updated second current pose, and a time step of the gradient descent algorithm; Wherein, determining the loss function includes: A fourth sub-loss function corresponding to the difference between the predicted pose and the updated second current pose and a fourth weight value corresponding to the fourth sub-loss function are determined.

5. The method for controlling a robotic arm according to any one of claims 1 to 3, wherein: The control method includes: Determining an updated second current pose of the end effector in world coordinates according to the updated first joint angle, the forward kinematics information, and the first current pose; Obtaining a velocity of a joint of the robotic arm according to the second current pose, the updated second current pose, and a time step of the gradient descent algorithm; determining a predicted joint angle of the robotic arm based on a velocity of a joint of the robotic arm, the updated first joint angle, and a time step of the gradient descent algorithm; Determining a predicted pose of the end effector in world coordinates according to the predicted joint angle of the robotic arm, the forward kinematics information, and a first current pose of the end effector; Wherein, determining the loss function includes: Determine a fifth sub-loss function corresponding to the difference between the predicted pose and the updated second current pose and a fifth weight value corresponding to the fifth sub-loss function.

6. The method for controlling a robotic arm according to any one of claims 1 to 5, wherein: The control method further includes: Obtaining a three-dimensional model of a target object, wherein the target object includes the robotic arm and an obstacle; Based on the three-dimensional model, a preset number of balls are generated to represent the surface contours of different regions of the target object, wherein the density of the balls in the different regions corresponds to the geometric complexity of the surface contours of the different regions.

7. The control method of the robot arm according to claim 6, characterized in that: The obstacle is a rigid object, and the loss function further includes a sixth sub-loss function corresponding to avoiding a collision between the robotic arm and the obstacle, and the sixth sub-loss function corresponds to a sixth weight value; Determining the loss function includes: Determine all first ball pairs based on the ball of the robotic arm and the ball of the obstacle, wherein the first ball pair is composed of any ball of the obstacle and any ball of the robotic arm; The sixth sub-loss function is determined based on a comparison between a spherical surface distance between two balls in the first ball pair and a preset threshold.

8. The method for controlling a robotic arm according to claim 6 or 7, wherein: The obstacle is a flexible object, and obtaining the three-dimensional model of the target object includes: Get data of real-time video stream; generating a three-dimensional model of the flexible object based on the two-dimensional image of the flexible object in the video stream; The loss function further includes a seventh sub-loss function corresponding to avoiding a collision between the robotic arm and the flexible object, and the seventh sub-loss function corresponds to a seventh weight value; Determining the loss function further includes: determining all second ball pairs based on the ball of the robotic arm and the ball of the flexible object, wherein the second ball pair is composed of any ball of the robotic arm and any ball of the flexible object; The seventh sub-loss function is determined based on a comparison between a spherical surface distance between two balls in the second ball pair and a preset threshold.

9. The method for controlling a robotic arm according to any one of claims 6 to 8, wherein: The loss function further includes an eighth sub-loss function corresponding to preventing the connecting rods of the robotic arm from colliding with each other, and the eighth sub-loss function corresponds to an eighth weight value; Determining the loss function includes: Based on the balls of the robotic arm, determining all third ball pairs, wherein the third ball pairs are composed of any two balls on different links of the robotic arm; The eighth sub-loss function is determined based on a comparison between a spherical surface distance between two balls in the third ball pair and a preset threshold.

10. The method for controlling a robotic arm according to any one of claims 6 to 9, wherein: The target object includes a physical surface within a workspace of the robotic arm, the loss function further includes a ninth loss function corresponding to avoiding a collision between the robotic arm and the physical surface, and the ninth sub-loss function corresponds to a ninth weight value; Determining the loss function includes: Determine all fourth ball pairs based on the ball of the robotic arm and the ball of the physical surface, wherein the fourth ball pair is composed of any ball of the robotic arm and any ball of the physical surface; The ninth sub-loss function is determined based on a comparison between a spherical surface distance between two balls in the fourth ball pair and a preset threshold.

11. The method for controlling a robotic arm according to any one of claims 1 to 10, wherein: The loss function further includes a tenth sub-loss function for limiting the angle range of the joint angle of the robotic arm, and the tenth sub-loss function corresponds to a tenth weight value; Determining the loss function includes: The tenth sub-loss function is determined based on a comparison between the first joint angle of the robotic arm and a preset joint angle threshold.

12. The method for controlling a robotic arm according to any one of claims 1 to 11, wherein: The loss function further includes an eleventh sub-loss function for limiting the speed of the end effector, and the eleventh sub-loss function corresponds to an eleventh weight value; Determining the loss function includes: Determining an updated second current pose of the end effector in world coordinates according to the updated first joint angle, the forward kinematics information, and the first current pose; Obtaining a second velocity of the end effector according to the second current pose, the updated second current pose, and a time step of the gradient descent algorithm; The eleventh sub-loss function is determined based on a comparison between the second speed of the end effector and a preset speed threshold, wherein the preset speed threshold includes a maximum speed threshold, and optionally, the preset speed threshold may also include a minimum speed threshold.

13. The method for controlling a robotic arm according to any one of claims 1 to 12, wherein: When the value of each sub-loss function of the loss function is lower than a predetermined threshold, the current posture of the robotic arm matches the target posture.

14. The method for controlling a robotic arm according to claim 13, wherein: The control method further includes: When the value of each sub-loss function of the loss function is lower than a predetermined threshold, a first instruction is received, where the first instruction instructs the end effector to perform an object grasping operation.

15. The method for controlling a robotic arm according to any one of claims 1 to 14, wherein: Determining the loss function includes: Obtaining status information of the robotic arm, wherein the status information indicates at least one of the following: whether the robotic arm holds an object; whether the robotic arm is moving; whether the robotic arm is stationary; Whether the current posture of the robotic arm matches the target posture; whether the robotic arm is grasping an object; whether the robotic arm is releasing the object it is holding; Adjusting the weight value corresponding to each sub-loss function of the loss function according to the state information; Optionally, determining the loss function may include: The state information is determined according to the instruction received by the robot arm and whether the end effector performs grasping of the object.

16. The method for controlling a robotic arm according to any one of claims 1 to 15, wherein: The control method further includes: When the end effector holds an object, a ball representing a surface profile of the end effector is stacked with a ball representing the object, wherein the stacked balls represent the object as an extension of the end effector to represent the surface profile of the extended end effector.

17. The method for controlling a robotic arm according to claim 16, wherein: The end effector is provided with a sensor, for example, a tactile sensor, and the control method further includes: receiving image information of all objects in the working space of the robotic arm and the robotic arm sent by an image acquisition module; Obtaining feedback information sent by the sensor; It is determined whether the end effector holds an object according to the image information and the feedback information.

18. The method for controlling a robotic arm according to claim 15, wherein: The control method further includes: receiving image information of all objects in the working space of the robotic arm and the robotic arm sent by an image acquisition module; Displaying the image information on a user interface of a display; and / or, The instruction is generated in response to a user operation on the user interface.

19. A control system for a robotic arm, characterized in that: The control system includes: one or more processors, and the one or more processors are used to execute a computer program to control the robotic arm to implement the control method according to any one of claims 1 to 16.

20. The control system of the robot arm according to claim 19, characterized in that: The control system includes: An image acquisition module is used to acquire image information of all objects in the workspace and the robotic arm, and send the image information to the one or more processors.

21. The control system of the robot arm according to claim 20, characterized in that: The control system further comprises: A robotic arm, wherein the robotic arm comprises a base, at least one link, and a joint connecting the at least one link, wherein the at least one link comprises an end effector; a sensor provided on the end effector of the robotic arm, for example, a tactile sensor; The one or more processors are further configured to: receiving image information of all objects in the working space of the robotic arm and the robotic arm sent by an image acquisition module; Obtaining feedback information sent by the sensor; It is determined whether the end effector holds an object according to the image information and the feedback information.

22. The control system of the robot arm according to claim 19, wherein: The control system further includes a display, which is used to: displaying the image information on a user interface; and / or, receiving a user operation through the user interface; The one or more processors are configured to generate the instructions based on the operations.

23. An electronic device, characterized in that: include: a memory storing a computer program; One or more processors, which can control the robotic arm to implement the control method according to any one of claims 1 to 18 when executing the computer program.

24. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by one or more processors, the robot arm can be controlled to implement the control method according to any one of claims 1 to 18.

25. A computer program product comprising a computer program, characterized in that When the computer program is executed by one or more processors, the robot arm can be controlled to implement the control method according to any one of claims 1 to 18.

Images