Control methods, systems, electronic devices, media, and software products for robotic arms

By employing gradient descent algorithm and loss function to optimize joint angles in the robotic arm control method, the problems of high computational cost and slow response speed of existing robot motion path control algorithms are solved, and efficient and accurate robotic arm motion planning is achieved.

CN120503205BActive Publication Date: 2026-11-10WENZHOU UNIVERSITY ARTIFICIAL INTELLIGENCE & ADVANCED MANUFACTURING INSTITUTE (YONGJIA)
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202510845023.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2025-06-23
Publication Date
2026-11-10
Estimated Expiration
2045-06-23

AI Technical Summary

Technical Problem

Existing robot motion path control algorithms suffer from high computational costs, insensitivity to rapid environmental changes, slow response speed, and high collision rates, especially in robotic arm applications. In particular, ML-based control algorithms have limitations in terms of efficiency and stability.

Method used

By using gradient descent algorithm to update joint angles in the control method of robotic arm, and utilizing loss function including multiple sub-loss functions and weight values, combined with positive kinematic information, the trajectory of end effector is optimized to avoid collisions, without the need to collect training data in advance.

Benefits of technology

It achieves optimized trajectory planning for the end effector of the robotic arm, reduces computational costs, improves control accuracy and efficiency, and enables rapid response to environmental changes while avoiding collisions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120503205B_ABST
    Figure CN120503205B_ABST
Patent Text Reader

Abstract

A control method, system, electronic device, medium, and program product for a robotic arm are disclosed. The control method includes: acquiring the target pose of the end effector in world coordinates and the forward kinematics information of the robotic arm; determining the second current pose of the end effector in world coordinates based on the first joint angle of the robotic arm, the forward kinematics information, and the first current pose of the end effector; determining a loss function, the loss function including a first loss function corresponding to the difference between the target pose and the second current pose; and updating the first joint angle using a gradient descent algorithm based on the loss function to control the movement of the robotic arm. This application eliminates the need for pre-collection of training data to control the movement of the robotic arm, reducing computational costs.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the field of robotics, and more particularly to a control method, system, electronic device, medium, and program product for a robotic arm. Background Technology

[0002] There are many types of control algorithms for robot motion paths, suitable for different types of robot systems and environments. Generally speaking, these control algorithms can be divided into two main categories: traditional algorithms and ML (Machine Learning) based algorithms.

[0003] Traditional algorithms mainly fall into four categories: graph search algorithms (e.g., Dijkstra's algorithm), sampling-based algorithms (e.g., Probabilistic Road Map, PRM), curve interpolation algorithms (e.g., spline curve algorithms), and reaction-based algorithms (e.g., Potential Field Method, PFM). ML-based control algorithms consist of classical ML algorithms (e.g., Random Forest), deep learning models such as CNNs (Convolutional Neural Networks), and RL (Reinforcement Learning) algorithms (e.g., Behavior-Judgment Algorithms and Proximal Policy Optimization Algorithms).

[0004] Graph search algorithms include best-first search, depth-first search, and breadth-first search. Best-first search computes heuristics (e.g., value or cost) for adding each node and edge to the search tree. Based on this, the search process is guided by the value gain in the search path to determine which branch to expand in the search tree. Depth-first search expands the search tree deeper and deeper using a priority strategy from the start point to the end point until a feasible path is found. Breadth-first search is similar to depth-first search, except that the expansion of the search tree is prioritized in terms of breadth. However, as image resolution increases, graph search algorithms become computationally expensive and insensitive to rapid environmental changes.

[0005] Sampling-based algorithms generate paths by randomly sampling a fixed workspace. To build a tree, the Rapidly-exploring Random Trees (RRT) algorithm attempts to explore the workspace uniformly through random search. The Probabilistic Roadmap Method (PRM) algorithm consists of two phases: a learning phase (preparing a collision-free probability roadmap) and a query phase (constructing a trajectory between the origin and the target based on the probability map). It's worth noting that sampling-based motion planning is slow and has a high collision rate due to its high initialization latency. These methods are also computationally expensive due to the large search space in the application.

[0006] Curve interpolation algorithms aim to interpolate a set of mathematical rules to create trajectories. For example, cyclotron curves, polynomial curves, and spline curves can be used as trajectory smoothing and curve generation patterns.

[0007] Unlike graph search algorithms, which spend a significant amount of time on global path planning, reaction-based algorithms aim to react to or control local conditions based on the current environment. Reaction-based algorithms are widely used in industrial applications, such as potential field methods and dynamic window methods. The potential field is a variable real-valued function U, whose value can be viewed as the cost of achieving the goal (e.g., the distance to the target position or pose), and its gradient can be viewed as velocity. Its inverse gradient... Depending on the robot's current information x (e.g., the robot's position), it is a vector representing the local direction of reducing U. Therefore, in such potential fields, the robot is driven to move along the anti-gradient of the potential field to minimize U. However, a drawback of methods like PFM is that it often gets trapped in local minima when the objective function is non-convex. Furthermore, when the robot moves at high speed between closely spaced obstacles, it experiences severe motion oscillations. In practical applications, some variants, such as the VFH (Vector Field Histogram) algorithm and the HPF (Harmonic Potential Field) algorithm, can partially alleviate these drawbacks.

[0008] Motion path control algorithms based on machine learning (ML) can be divided into two categories: classical ML algorithms and logic-based (RL) algorithms. In classical ML algorithms, CNNs significantly improve the ability to extract high-level features from images. While commonly used in image-related tasks, including motion planning, they don't actually consider any temporal or sequential factors in the problem. Neural networks combined with LSTM (Long Short-Term Memory) or Markov chain-based algorithms (such as Q-learning) can achieve temporal motion planning. However, they have limitations in efficiency, such as slow network convergence and high training data requirements. Google DeepMind's Natural DQN (Deep Q-Learning) partially addresses this issue by reusing old data in the response buffer to improve efficiency. To further improve robustness when there are biases in state-action value (i.e., Q-value) estimation, Double DQN and Dueling DQN have been proposed to handle noise-induced problems. On the other hand, policy gradient algorithms have higher stability in network convergence, but their convergence speed is limited. To address this problem, behavior-criteria algorithms were invented, which improve convergence speed by adding a criterion structure to update state value estimates based on improved policies. Later, various methods were proposed, such as A2C (Advantage Actor-Critic), PPO (Proximal Policy Optimization), TRPO (Trust Region Policy Optimization), and DDPG (Deep, Deterministic Policy Gradient), to improve convergence stability while maintaining faster convergence speed. However, implementing these learning-based algorithms still requires significant computational resources and data or environment setup. Summary of the Invention

[0009] This disclosure provides a control method for a robotic arm, the robotic arm including a base, at least one link, and a joint connecting the at least one link, wherein the at least one link includes an end effector, and the control method includes:

[0010] Obtain the target pose of the end effector in world coordinates and the positive kinematics of the robotic arm;

[0011] Based on the first joint angle of the robotic arm, the information of the forward kinematics, and the first current pose of the end effector in the coordinate system of the joint connected to the end effector, the second current pose of the end effector in world coordinates is determined.

[0012] Determine a loss function, the loss function including a first loss function corresponding to the difference between the target pose and the second current pose;

[0013] Based on the loss function, the first joint angle is updated using a gradient descent algorithm to control the movement of the robotic arm.

[0014] Optionally, the target pose includes the coordinates of a first three-dimensional grasping point, a first grasping direction vector, and a first grasping direction normal vector;

[0015] The second current pose includes the coordinates of the second three-dimensional grasping point, the second grasping direction vector, and the second grasping direction normal vector.

[0016] Optionally, the first loss function includes:

[0017] The first sub-loss function corresponds to the difference between the coordinates of the first sub-3D grasping point and the coordinates of the second sub-3D grasping point, and the first sub-loss function corresponds to the first weight value.

[0018] The second sub-loss function corresponds to the difference between the grasping direction vector and the second grasping direction vector, and the second sub-loss function corresponds to the second weight value;

[0019] The third sub-loss function corresponds to the difference between the grasping direction normal vector and the second grasping direction normal vector, and the third sub-loss function corresponds to the third weight value.

[0020] Optionally, the control method includes:

[0021] Based on the updated first joint angle, the information of the positive kinematics, and the first current pose, the updated second current pose of the end effector in world coordinates is determined;

[0022] Based on the second current pose, the updated second current pose, and the time step of the gradient descent algorithm, the first velocity of the end effector is obtained;

[0023] Based on the first velocity of the end effector, the updated second current pose, and the time step of the gradient descent algorithm, the predicted pose of the end effector in world coordinates is determined.

[0024] The loss function to be determined includes:

[0025] Determine a fourth sub-loss function corresponding to the difference between the predicted pose and the updated second current pose, and a fourth weight value corresponding to the fourth sub-loss function.

[0026] Optionally, the control method includes:

[0027] Based on the updated first joint angle, the information of the positive kinematics, and the first current pose, the updated second current pose of the end effector in world coordinates is determined;

[0028] Based on the second current pose, the updated second current pose, and the time step of the gradient descent algorithm, the speed of the joints of the robotic arm is obtained;

[0029] Based on the speed of the robotic arm's joints, the updated first joint angle, and the time step of the gradient descent algorithm, the predicted joint angle of the robotic arm is determined.

[0030] Based on the predicted joint angles of the robotic arm, the information of the forward kinematics, and the first current pose of the end effector, the predicted pose of the end effector in world coordinates is determined.

[0031] Determining the loss function includes:

[0032] A fifth sub-loss function and a fifth weight value corresponding to the difference between the predicted pose and the updated second current pose are determined.

[0033] Optionally, the control method further includes:

[0034] Obtain a 3D model of the target object;

[0035] Based on the three-dimensional model, a preset number of spheres are generated to represent the surface contours of different regions of the target object, wherein the density of spheres in different regions corresponds to the geometric complexity of the surface contours of the different regions.

[0036] Optionally, the target object includes the robotic arm and the obstacle, and the loss function further includes a sixth sub-loss function corresponding to avoiding collision between the robotic arm and the obstacle, the sixth sub-loss function corresponding to a sixth weight value;

[0037] The determination of the loss function also includes:

[0038] Based on the ball in the robotic arm and the ball in the obstacle, all first ball pairs are determined, wherein each first ball pair consists of any one ball from the obstacle and any one ball from the robotic arm;

[0039] The sixth sub-loss function is determined by comparing the spherical distance between the two balls in the first ball pair with a preset threshold.

[0040] Optionally, the obstacle may include a flexible object, and obtaining the three-dimensional model of the target object includes:

[0041] Obtain data from the real-time video stream;

[0042] A three-dimensional model of the target object is generated based on the two-dimensional image of the flexible object in the video stream.

[0043] The loss function further includes a seventh sub-loss function that corresponds to avoiding collisions between the robotic arm and the flexible object, and the seventh sub-loss function corresponds to a seventh weight value.

[0044] The determination of the loss function also includes:

[0045] Based on the ball of the robotic arm and the ball of the flexible object, all second ball pairs are determined, wherein the second ball pair consists of any one ball of the robotic arm and any one ball of the flexible object;

[0046] The seventh sub-loss function is determined by comparing the spherical distance between the two balls in the second ball pair with a preset threshold.

[0047] Optionally, the target object includes the robotic arm, and the loss function further includes an eighth sub-loss function that corresponds to avoiding collisions between the links of the robotic arm, and the eighth sub-loss function corresponds to an eighth weight value;

[0048] The determined loss function includes:

[0049] Based on the balls of the robotic arm, determine all third ball pairs, wherein the third ball pair consists of any two balls on different links of the robotic arm;

[0050] The eighth sub-loss function is determined by comparing the spherical distance between the two balls in the third ball pair with a preset threshold.

[0051] Optionally, the target object includes the robotic arm and the physical surface within the workspace of the robotic arm. The loss function further includes a ninth loss function corresponding to avoiding collisions between the robotic arm and the physical surface, and the ninth sub-loss function corresponds to a ninth weight value. Determining the loss function includes: determining all fourth ball pairs based on the balls on the robotic arm and the balls on the physical surface, wherein each fourth ball pair consists of any one ball from the robotic arm and any one ball from the physical surface; and determining the ninth sub-loss function based on a comparison of the spherical distance between the two balls in the fourth ball pair with a preset threshold.

[0052] Optionally, the loss function further includes a tenth sub-loss function for limiting the angle range of the joint angles of the robotic arm, the tenth sub-loss function corresponding to a tenth weight value;

[0053] The determination of the loss function includes: determining the tenth sub-loss function based on a comparison between the first joint angle of the robotic arm and a preset joint angle threshold.

[0054] Optionally, the loss function further includes an eleventh sub-loss function for limiting the speed of the end effector, the eleventh sub-loss function corresponding to an eleventh weight value;

[0055] The determined loss function includes:

[0056] Based on the updated first joint angle, the information of the forward kinematics, and the first current pose, the updated second current pose of the end effector in world coordinates is determined; based on the second current pose, the updated second current pose, and the time step of the gradient descent algorithm, the second velocity of the end effector is obtained; based on the comparison between the second velocity of the end effector and a preset velocity threshold, the eleventh sub-loss function is determined, wherein the preset velocity threshold includes a maximum velocity threshold, and optionally, the preset velocity threshold may also include a minimum velocity threshold.

[0057] Optionally, when the value of each sub-loss function of the loss function is lower than a predetermined threshold, the current pose of the robotic arm matches the target pose.

[0058] Optionally, the control method further includes: receiving a first instruction when the value of each sub-loss function of the loss function is lower than a predetermined threshold, the first instruction instructing the end effector to perform an object grasping operation.

[0059] Optionally, the control method further includes: determining the state information of the robotic arm based on the control command received by the robotic arm, whether the target object and the end effector perform object grasping, and adjusting the number, type, and corresponding weight value of the sub-loss functions in the loss function based on the state information of the robotic arm.

[0060] Optionally, determining the loss function includes: obtaining the state information of the robotic arm, wherein the state information indicates at least one of the following: whether the robotic arm is holding an object; whether the robotic arm is moving; whether the robotic arm is stationary; whether the current pose of the robotic arm matches the target pose; whether the robotic arm is grasping an object; whether the robotic arm is releasing the held object; adjusting the weight value corresponding to each sub-loss function of the loss function according to the state information; optionally, determining the loss function may include: determining the state information based on the instructions received by the robotic arm and whether the end effector performs object grasping.

[0061] Optionally, the control method further includes: when the end effector holds an object, stacking a ball representing the surface profile of the end effector with a ball representing the object, wherein the stacked balls represent the object as an extension of the end effector to represent the surface profile of the extended end effector.

[0062] Optionally, the end effector is equipped with a sensor, such as a tactile sensor, and the control method further includes:

[0063] Receive image information of all objects in the workspace of the robotic arm and the robotic arm itself, sent by the image acquisition module;

[0064] Obtain the feedback information sent by the sensor;

[0065] The end effector is determined to hold an object based on the image information and the feedback information.

[0066] Optionally, the control method further includes:

[0067] Receive image information of all objects in the workspace of the robotic arm and the robotic arm itself, sent by the image acquisition module;

[0068] Displaying the image information on the graphical user interface of the display; and / or,

[0069] The instructions are generated in response to user actions on the graphical user interface.

[0070] This disclosure provides a control system for a robotic arm, the control system comprising: one or more processors, the one or more processors being configured to execute a computer program to control the robotic arm to implement the aforementioned control method.

[0071] Optionally, the control system further includes an image acquisition module, which acquires image information of all objects in the workspace and the robotic arm, and sends the image information to the one or more processors.

[0072] Optionally, the control system further includes:

[0073] A robotic arm, wherein the robotic arm includes a base, at least one link and a joint connecting the at least one link, wherein the at least one link includes an end effector;

[0074] Sensors, such as tactile sensors, are mounted on the end effector of the robotic arm.

[0075] The one or more processors are also used for:

[0076] Receive image information of all objects in the workspace of the robotic arm and the robotic arm itself, sent by the image acquisition module;

[0077] Obtain the feedback information sent by the sensor;

[0078] The end effector is determined to hold an object based on the image information and the feedback information.

[0079] Optionally, the control system further includes a display, wherein the display is configured to: display the image information processed by the one or more processors on the user interface; and / or receive user operations through the user interface;

[0080] The one or more processors are used to generate the instructions according to the operation.

[0081] This disclosure provides an electronic device including a memory, one or more processors, and a computer program stored in the memory and used to run on the processors, wherein the one or more processors execute the computer program to implement the aforementioned control method for a robotic arm.

[0082] This disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by one or more processors, implements the aforementioned control method for a robotic arm.

[0083] This disclosure provides a computer program product, including a computer program that, when executed by one or more processors, implements the aforementioned control method for a robotic arm.

[0084] This disclosure, in some embodiments, uses a loss function based on the difference between the target pose and the current pose of the robotic arm in the same coordinate system, and employs a gradient descent algorithm to iteratively update the joint angles of the robotic arm, thereby achieving control of the robotic arm. This robotic arm control method can generate an optimized trajectory for the end effector of the robotic arm, guiding the end effector towards the target pose, and does not require pre-collection of training data, thus reducing computational costs. Attached Figure Description

[0085] To more clearly illustrate the technical solutions of the embodiments in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some examples or embodiments of this specification. For those skilled in the art, these drawings can be applied to other similar scenarios without creative effort.

[0086] Figure 1 This is a flowchart illustrating a control method for a robotic arm provided in Embodiment 1 of this disclosure.

[0087] Figure 2 A schematic diagram of a simulated robotic arm provided in some embodiments of this disclosure is shown.

[0088] Figure 3 A schematic diagram of a physical robotic arm provided in some embodiments of this disclosure is shown.

[0089] Figure 4 It shows a Model predictive control A schematic diagram illustrating the algorithm's principle.

[0090] Figure 5a A flowchart illustrating a control method for a robotic arm provided in some embodiments of this disclosure is shown.

[0091] Figure 5b A schematic flowchart of a method for determining a loss function provided in some embodiments of this disclosure is shown.

[0092] Figure 5c A flowchart illustrating a control method for a robotic arm provided in some embodiments of this disclosure is shown.

[0093] Figure 5d A schematic flowchart of a method for determining a loss function provided in some embodiments of this disclosure is shown.

[0094] Figure 6 This disclosure illustrates a method for using spheres to represent the surface contours of different regions of a target object, as provided in some embodiments.

[0095] Figure 7 A schematic diagram of a neighborhood provided in some embodiments of this disclosure is shown.

[0096] Figure 8 This illustration shows an example of constructing a 3D model of a hand in real time from camera images, as provided in some embodiments of this disclosure.

[0097] Figure 9 The diagram shows some examples represented using colliding spheres.

[0098] Figure 10 This illustration shows a schematic diagram of a process by which a gripper at the end of a robot grasps a target object, as provided in some embodiments of this disclosure.

[0099] Figure 11 A schematic flowchart of a method for determining a loss function provided in some embodiments of this disclosure is shown.

[0100] Figure 12 This diagram illustrates the transition relationships between states of a robotic arm provided in some embodiments of this disclosure.

[0101] Figure 13 This illustration shows a schematic diagram of the interaction process between a robotic arm and an object during a grasping-placement operation, as provided in some embodiments of this disclosure.

[0102] Figure 14 A graphical user interface for user control of a robotic arm is shown in some embodiments of this disclosure.

[0103] Figure 15 A schematic diagram of a control system for a robotic arm provided in some embodiments of this disclosure is shown.

[0104] Figure 16 A schematic diagram of a multithreaded system architecture provided in some embodiments of this disclosure is shown.

[0105] Figure 17 A schematic diagram of the structure of an electronic device provided in some embodiments of this disclosure is shown. Detailed Implementation

[0106] The present invention will be further illustrated by way of embodiments below, but the present invention is not limited to the scope of the embodiments described herein.

[0107] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the document does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0108] It should be understood that the terms “device,” “system,” “unit,” and / or “module” used herein are one way to distinguish different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0109] As illustrated herein, unless the context clearly indicates otherwise, the words “a,” “an,” “an,” and / or “the” do not specifically refer to the singular and may also include the plural. Generally speaking, the terms “comprising” and “including” only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0110] The definitions used herein, such as the terms “having,” “may have,” “comprising,” or “may include,” indicate the presence of the corresponding function, operation, element, etc., and do not limit the presence of one or more other functions, operations, elements, etc. Furthermore, it should be understood that the terms “comprising” or “having” as used herein indicate the presence of the features, figures, steps, operations, elements, components, or combinations thereof described in the specification, without excluding the presence or addition of one or more other features, figures, steps, operations, elements, components, or combinations thereof.

[0111] Flowcharts are used in this document to illustrate the operations performed by the system according to the embodiments herein. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0112] In some examples of this disclosure, the method steps described below can be implemented by a program or custom circuitry, or a combination of custom circuitry and a program. For example, the method steps can be executed by a combination of one or more GPUs, one or more CPUs, or any other technically feasible processor such as an ASIC chip. In some examples of this disclosure, one or more processors such as an ASIC chip may include memory. Furthermore, those skilled in the art will understand that the scope of this disclosure can include any system and apparatus capable of performing the method steps described below.

[0113] Example 1

[0114] This disclosure provides a method for controlling a robotic arm. This method allows for the planning of the robotic arm's motion trajectory.

[0115] In some examples of this disclosure, the robot may be a robotic arm. The robotic arm includes a base, at least one link, and a joint connecting the at least one link, wherein the at least one link includes an end effector. The joint can generate lateral and rotational movements for the robotic arm, thereby causing the end effector of the robotic arm to reach a designated position.

[0116] like Figure 1 As shown, the control method includes:

[0117] S110. Obtain information on the target pose of the end effector in world coordinates and the positive kinematics of the robotic arm.

[0118] In some examples of this disclosure, the origin of the robot base coordinate system is the same as the origin of the world coordinate system, that is, the target pose is the pose of the end effector in the robot base coordinate system.

[0119] In the examples disclosed herein, the target pose can be represented using, for example, three vector parameters: and in, Let be a vector representing the position of the target pose. Let the vector represent the direction of the target pose. This is a vector representing the normal direction of the target pose. Those skilled in the art will understand that the target pose is not limited to being represented by the above three vector parameters; it can also be represented in other ways, such as using a matrix.

[0120] Forward kinematics

[0121] Forward kinematics is a series of physical transformations that describe the physical transformations available to each component in a robot. The physical pose of a robot component is determined not only by the joint angles or translational phases (relative poses to adjacent components) of its directly connected joints, but also by the relative poses of those adjacent components relative to their own adjacent components. Therefore, it is more direct to describe the physical transformations of a robot component as a series of sequential transformations. In computer graphics, this composite transformation can be simply described as the product of the component's geometric point cloud coordinate grid and a 4x4 matrix. Typically, a unique 4x4 matrix is ​​provided for each individual component of the robot to describe the complete forward kinematics.

[0122] In forward kinematics, each joint of the robotic arm has a coordinate system. For example, the coordinate system can be set up as follows: for each coordinate system, the z-axis is aligned with the joint's axis, the x-axis points towards the next joint and is perpendicular to the z-axis of the previous joint's coordinate system and the current coordinate system's z-axis. The direction of the y-axis is then determined using the right-hand rule. Rotation and translation occur between adjacent coordinate systems, which can be described by the DH (Denavit-Hartenberg) parameter.

[0123] The DH parameters consist of four basic elements: θ (Theta), d (Distance), a (Link Length), and α (Twist Angle). Here, θ represents the rotation angle about the previous z-axis, d represents the translation distance along the previous z-axis, a represents the length of the common perpendicular from the previous z-axis to the current z-axis, and α represents the angle between the two z-axis. The DH parameter table defines the position and orientation of each joint of the robot through a series of values ​​for these four parameters, thus describing the pose of the entire robotic arm. For example, a planar robotic arm with two joints might have a DH parameter table containing two rows, each with four parameter values. These values ​​determine the position and orientation of each joint relative to the previous joint, ultimately determining the position and orientation of the end effector. The end effector could be, for example, a gripper.

[0124] Information about positive kinematics can be obtained, for example, by inputting a URDF (United Robotics Description Format) file.

[0125] Information from forward kinematics can be used to transform the pose from the end effector coordinate system to the robot base coordinate system.

[0126] URDF file

[0127] The Unified Robot Description Format (URDF) file is an example of a data format used to describe the physical geometry of a robot and its components, as well as the robot's degrees of freedom, in the form of a point cloud mesh, to enable physical structural transformations such as joint angle rotation and physical translation. The mesh components in an URDF file may include a visual mesh and a collision mesh; the visual mesh is used for visualizing a large number of 3D polygons, while the collision mesh is used for efficiently detecting a smaller number of 3D polygons.

[0128] S120. Based on the first joint angle of the robotic arm, the information of the forward kinematics, and the first current pose of the end effector in the coordinate system of the joint connected to the end effector, determine the second current pose of the end effector in the world coordinate system.

[0129] The first current pose can be the pose of the end effector in the coordinate system of the joint to which it is connected. In one example, the end effector can point to the Z-axis. Figure 2 A simulated robotic arm 210 is shown, which includes an end effector 214. The first current pose of the end effector 214 can be represented by three vector parameters: and in, Let the position vector represent the first current pose. Let the vector represent the direction of the first current pose. Let be the vector representing the normal direction of the first current pose. Based on these vector parameters, the first current pose in the coordinate system of the last joint of the robot arm can be transformed into a second current pose in the robot base coordinate system. This can be used to evaluate the alignment between the end effector pose and the target pose.

[0130] The following can be explained with a specific example. and The meaning of .

[0131] This indicates that the end effector is located at a point 15 cm away from the origin along the positive z-axis.

[0132] This indicates that the direction of the end effector is along the z-axis;

[0133] Or [-1,0,0], indicating that the direction of the normal to the end effector is in the positive or negative x-axis direction.

[0134] Figure 3 An example of a physical robotic arm 230 is shown, which includes an end effector 234. When the robot's task is to grasp a target object 238, the target pose is the pose required for the end effector 234 to grasp the target object 238; that is, the target grasping pose. The target grasping pose can also be represented by three vector parameters: and in, Let the vector represent the position point of the target's grasping pose. Let the vector represent the direction of the target grasping pose. A vector representing the normal direction of the target pose.

[0135] To achieve precise grasping of the target object 238, the target grasping pose of the end effector 234 relative to the target object must first be determined. This target grasping pose relative to the target object 238 provides the relative position and orientation of the end effector in the coordinate system of the target object. For example, this target grasping pose relative to the target object can be obtained directly through user input or through a grasping pose generation algorithm. The grasping pose generation algorithm can be based on a 4×4 transformation matrix P. G2O It indicates. P G2O Transform the static pose of the end effector into the required coordinate system of the target object.

[0136] Assuming the robot's base is located at the origin of the world coordinate system, the world coordinate system coincides with the robot's base coordinate system. Therefore, after transforming the pose to the world coordinate system, there is no need to transform it back to the robot's base coordinate system. Thus, in obtaining the 6D pose of the target object... Then, the target grasp pose in the target object's coordinate system can be transformed to the world coordinate system, thus obtaining the target grasp pose in the world coordinate system. The specific formula is as follows:

[0137]

[0138] in, A vector representing the position of the target's grasping pose; A vector representing the direction of the target grasping pose; The vector representing the normal direction of the target pose; Let be the vector representing the position of the first current pose; Let the vector represent the direction of the first current pose; P is a vector representing the normal direction of the first current pose; 6D P represents the 6D pose of the target object. G2O Let be the transformation matrix.

[0139] The target grasping pose in the world coordinate system was obtained. and This allows control over the pose of the end effector, gradually bringing it closer to the target grasping pose. Given initial values ​​θ for the robot's joint angles. ′ and target grasping posture and The goal is then to generate a smooth joint angle trajectory Tj = {θ} n |n∈{1,2,…N}} can be implemented by a robot; where N is the number of steps.

[0140] To guide the end effector toward the target pose, the forward kinematic transformation matrix FK(θ) is used to... and Transform to the robot base coordinate system to obtain the current pose of the end effector in the robot base coordinate system. The formula for this transformation is as follows:

[0141]

[0142] in, Let be the vector representing the position of the second current pose; Let the vector represent the direction of the second current pose; Let be the vector representing the normal direction of the second current pose; Let be the vector representing the position of the first current pose; Let the vector represent the direction of the first current pose; Let FK(θ) be the vector representing the normal direction of the first current pose; FK(θ) is the positive kinematic function with θ as the variable.

[0143] S130. Determine a loss function, which includes a first loss function corresponding to the difference between the target pose and the second current pose. The loss function can also be understood as a cost function. In the examples of this disclosure, the first loss function can correspond to the deviation (i.e., difference) in position and orientation between the current pose and the target pose. By using information from the differentiable forward kinematics of the robotic arm and the gradient descent algorithm, the deviation in position and orientation between the current pose and the target pose of the end effector can be minimized iteratively. This allows for the calculation of a set of joint angles for the robotic arm, enabling the robotic arm to move smoothly to the target pose.

[0144] S140. Based on the objective function, the angle of the first joint is updated using the gradient descent algorithm to control the movement of the robotic arm.

[0145] In the examples disclosed herein, since the forward kinematic function FK(θ) is differentiable, the objective function can be minimized using gradient descent; that is, the value of θ can be adjusted iteratively to reduce the loss. The update rule for θ in the t-th iteration is defined by the following formula:

[0146]

[0147] Where, θ t k is the joint angle at time step t; t The step size for each θ can be determined based on the optimization algorithm used; θ t+1 The joint angle at time step t+1; Let be the gradient of the objective function Loss with respect to the parameter θ at the t-th iteration. For example, k t It can be a scalar value used in PyTorch optimization algorithms.

[0148] This optimization process iterates continuously. Each update of the joint angle θ generates a smooth trajectory, gradually reducing the deviation between the current pose and the target pose of the end effector in position and direction. In the examples of this disclosure, when the value of the objective function is lower than a predetermined threshold, the pose of the end effector can be considered close to the target pose, and the iteration can be terminated at this point. Therefore, in the examples of this disclosure, by controlling the robotic arm to run along the calculated trajectory, the end effector can be made to reach the target pose, for example, for... Figure 2 The simulated robotic arm 210 shown can enable the end effector 214 to reach the target pose.

[0149] loss function

[0150] In the examples disclosed herein, the loss function may consist of one or more sub-loss functions and weight coefficients corresponding to each sub-loss function. The weight coefficients can also be understood as weight values. In the examples disclosed herein, the sub-loss function includes a first loss function, which represents the difference between the second current pose and the target pose.

[0151] For example, individual sub-loss functions are combined with their corresponding weight coefficients to form the overall loss function, as shown below:

[0152] Loss = λ1L1 + λ2L2 + ... + λ i L i

[0153] Where Loss is the loss function; L1 is the i-th sub-loss function; λ i These are the weight coefficients corresponding to the i-th sub-loss function. For example, the weight coefficients of each sub-loss function can be defined in a pre-defined manner. In the examples disclosed herein, each weight coefficient can also be further adjusted individually.

[0154] In some examples of this disclosure, the first loss function includes at least one of a position point loss function, a direction loss function, and a normal direction loss function. The position point loss function represents the error between the second current pose and the target pose at the position point, the direction loss function represents the error between the second current pose and the target pose in the direction vector, and the normal direction loss function represents the error between the second current pose and the target pose in the normal direction.

[0155] In the examples disclosed herein, the position point loss function can represent the difference between the current position of the end effector and the position of the target pose using the square of the Euclidean distance. For example, the formula is shown below:

[0156]

[0157] Among them, L dist For the objective function of the location point, This refers to the current pose and position of the end effector. Let be the location point of the target pose. The inventors discovered that using the square of the Euclidean distance allows for an intuitive distance measurement in 3D space, and that it is differentiable.

[0158] In the examples disclosed herein, the directional loss function uses cosine similarity to represent the difference between the orientation of the current pose of the end effector and the orientation of the target pose. For example, the formula is shown below:

[0159]

[0160] Among them, L dir The objective function is the direction. A vector representing the direction of the current pose of the end effector; Let be a vector representing the orientation of the target pose. The inventors discovered that using cosine similarity can achieve an intuitive measurement of the vector angle difference in 3D space, and that this measurement is differentiable.

[0161] In the examples disclosed herein, the normal direction sub-loss function is used to describe the difference between the normal direction of the current pose of the end effector and the normal direction of the target pose. When the end effector is symmetric, the absolute value of the cosine similarity can be used to constrain this. For example, the formula is shown below:

[0162]

[0163] Among them, L n_dir The objective function is the function along the normal direction. The vector representing the normal direction of the current pose of the end effector; Let be the vector representing the normal direction of the target pose. The inventors discovered that using the absolute value of cosine similarity offers flexibility, allowing the normal vector to be aligned with or opposite to the target vector.

[0164] In some examples of this disclosure, the sub-loss function may further include at least one of a velocity-limiting loss function and an angle-limiting loss function. The angle-limiting objective function can be used to limit the angle range of the end effector, while the velocity-limiting objective function can be used to limit the velocity range of the end effector.

[0165] Adding an angle-limiting loss function to the loss function helps ensure that the robot operates within the mechanical limitations of the robotic arm, thereby avoiding potential damage or instability during the robotic arm's movement.

[0166] For example, the angle-constrained loss function can be expressed by the following formula:

[0167]

[0168] Among them, L angle-limit (a) is the objective function with angle constraints; a is the joint angle in radians; a low This is the lower limit of a; a high is the upper limit of a; C is the scaling constant, which determines the severity of the penalty.

[0169] The speed limiting loss function is used to penalize links that exceed a specified speed range, thereby limiting the speed of the end effector during motion. Including a speed limiting loss function in the overall loss function offers several advantages:

[0170] 1. Limiting the maximum speed helps improve the safety of the robotic arm during operation.

[0171] The end effector is usually the fastest moving part of a robotic arm. Limiting the physical speed of the end effector in trajectory planning methods based on gradient descent helps improve the safety of human-machine interaction.

[0172] 2. Limiting the minimum speed helps to escape the local minimum of the loss function.

[0173] Although the robotic arm trajectory planning method in this embodiment can achieve low-level objectives while satisfying multiple constraints, optimization-based methods are still prone to getting trapped in local minima of the loss function, especially when the obstacles in the robot's environment have complex geometries, such as large obstacles with a high aspect ratio. In this case, a lower end-effector speed limit may force the robot to continue moving, eventually escaping the local minima of the loss function while also avoiding collisions between the robotic arm and obstacles in the environment.

[0174] For example, suppose the forward kinematics function corresponding to the end effector is FK. ee (θ t If the time Δt is spent between the execution of consecutive robot control commands, then the velocity limitation loss function can be expressed by the following formula:

[0175]

[0176] Among them, FK ee (θ t ) represents the positive kinematics function of the end effector; Let L be the vector representing the position of the first current pose; speed-limit (v ee,t ) is the speed limiting objective function; v lim,min (t) represents the lower limit of the velocity, and is the value that changes with time; v represents the position of the end effector at time step t; eef,t V represents the speed at which the final execution occurs at time step t. max is the upper limit of speed, and is a constant value; C is a scaling constant that determines the severity of the penalty.

[0177] In this part of the disclosure, v lim,min(t) is a curve that changes over time, starting from a positive value of, for example, 10 cm / s, and decreasing to 0 cm / s over time.

[0178] Each sub-loss function in the loss function penalizes deviations from the desired pose, orientation, and spatial constraints, such as joint angle limitations. Using the loss function, θ can be updated to generate the robot arm's motion trajectory via gradient descent. This process continues to generate updated θ until the value of the loss function falls below a predetermined threshold, indicating that the end effector's current pose is very close to the target pose.

[0179] Model Predictive Control (MPC) Algorithm

[0180] MPC (Multi-Purpose Control) algorithms are powerful control techniques that optimize future control actions within a finite timeframe, taking into account system dynamics and any constraints. Because of their effectiveness in handling multivariable systems, nonlinearities, and constraints, this method has found widespread application in robotics.

[0181] The MPC algorithm optimizes the control input at each time step by solving the following optimization problem:

[0182]

[0183] Where t is the current time step; is the predicted output at time step t+k; w(k) is the reference trajectory; u(k) is the control input at time step t+k; Q and R are weight matrices; N is the prediction range; M is the control range.

[0184] The principle of the MPC algorithm is as follows: Figure 4 As shown, the optimization problem can minimize the error between the predicted state and the reference state while keeping the control input within a feasible range.

[0185] The inventors discovered that the limitations of the MPC algorithm include:

[0186] Computational complexity: Solving the optimization problem at each time step can consume a lot of computational resources, especially for large systems with multiple degrees of freedom.

[0187] Model accuracy: Performance depends on the accuracy of the system model. Any error between the model and the actual robot behavior can lead to suboptimal control operations.

[0188] Real-time limitations: Real-time applications require rapid computation, which may limit the complexity of the model and the range of predictions.

[0189] The inventors also discovered that the advantages of the MPC algorithm include:

[0190] Optimality: MPC optimizes control within a limited range, ensuring optimal performance under model and constraint conditions.

[0191] Handling constraints: This naturally includes restrictions on system state and inputs, such as joint constraints, speed constraints, and safety constraints.

[0192] Predictability: MPC can predict future system states, making it well-suited for dynamic and uncertain environments.

[0193] Loss function based on MPC algorithm

[0194] Joint angle prediction using joint velocity, forward and inverse kinematics (IK-MPC)

[0195] The inventors discovered that the MPC algorithm can optimize the joint velocity at each time step by solving the following optimization problem:

[0196]

[0197] Where t is the current time step, and q(k) is the predicted joint angle at time step t+k; q ik (k) represents the reference joint angle obtained based on inverse kinematics; Δq(k) is the control signal, which in this example is the joint velocity at time step t+k; q min and q max The joint angle is limited and determined by the robot's physical capabilities; Δq min and Δq max To control input limits, namely the minimum and maximum allowed joint velocities.

[0198] The inventors discovered that the main difference between this algorithm and joint angle prediction-based algorithms lies in the fact that the inverse kinematics solution is used as a reference for the MPC algorithm. In this algorithm, optimization can be divided into two sub-problems:

[0199] (1) MPC is used as a feedforward to optimize joint speed using inverse kinematics and guide the robot from the initial joint angle to the target angle configuration.

[0200] (2) Using gradient-optimized differentiable positive kinematics, the joint angles predicted by MPC are added to the optimized joint angles. The sum of the two control signals makes the robot move faster.

[0201] The advantages of this algorithm are as follows:

[0202] Separate control: MPC does not interfere with gradient-based feedback optimization, and the two optimizations are performed simultaneously.

[0203] Combined speed control: MPC can limit the control signal according to user needs.

[0204] The inventors discovered that MPC can be viewed as a method for pre-planning trajectories within a certain prediction range, while ignoring potential collisions with objects in the system. This makes MPC unsuitable for real-time reaction control in dynamic environments with multiple constraints (collisions). However, combining the step size generated by MPC with gradient control based on a loss function can further optimize the robot's motion trajectory. Assuming that MPC generates a set of u(0) inputs, this input value can be used as a base value and modified using the aforementioned gradient control-based loss function, as shown in the following formula:

[0205] θ t =q(0)+α t

[0206]

[0207] Where, k t The step size for each joint angle is determined by the optimization algorithm used. Let be the gradient of the loss function Loss at time step t with respect to the parameter α.

[0208] In some examples of this disclosure, after considering the principles, advantages and disadvantages of the two MPC algorithms mentioned above, the inventors found that the following two algorithms can be used to further improve the gradient-based robotic arm joint angle optimization process of this disclosure by using predictions based on future vision.

[0209] 1. Using coordinate velocity to predict the future position of the end effector

[0210] In this algorithm, the velocity of the end effector at the previous time step can be used to predict its position in the next few time steps. Correspondingly, such as... Figure 5a As shown in this partial example of the disclosure, the method 100 for controlling the robotic arm may include:

[0211] S252. Determine the updated second current pose of the end effector in world coordinates. As described above, based on the updated first joint angles, the information of the forward kinematics, and the first current pose, the pose of the end effector at time step t in world coordinates can be determined, which is the updated second current pose. From the pose of the end effector at time step t, its position p(t) at time step t can be determined.

[0212] S254. Obtain the first velocity of the end effector. For example, the first velocity is the velocity of the end effector in the previous time step. As mentioned above, this can be obtained based on the second current pose of the previous time step, the pose at time step t (i.e., the updated second current pose), and the time step k of the gradient descent algorithm. t , to obtain the first speed of the end effector.

[0213] S256. Determine the predicted pose of the end effector in world coordinates. For example, based on the end effector's first velocity, the updated second current pose, and the time step of the gradient descent algorithm, the predicted pose of the end effector in world coordinates, i.e., the pose at the next time step, can be determined. Assuming the end effector's position at time step t is p(t) and its velocity at time step t is v(t), a simple first-order approximation algorithm can be used to predict the end effector's position at the next time step t+1. For example, the formula can be as follows:

[0214] p(t+1)=p(t)+v(t)Δt

[0215] Where p(t) is the position of the end effector at time step t; v(t) is the velocity of the end effector at time step t; and Δt is the time step size. The time step size Δt can be equal to the time step size of the gradient descent algorithm. The time step size Δt can also be different from the time step size of the gradient descent algorithm. For example, the time step size Δt can be equal to the time step sizes of several gradient descent algorithms.

[0216] like Figure 5b As shown, determining the loss function S130 may include:

[0217] S260. Determine the fourth sub-loss function corresponding to the difference between the predicted pose and the updated second current pose, and the fourth weight value corresponding to the fourth sub-loss function.

[0218] By including a fourth sub-loss function and its weights in the loss function, the predicted positions can be used to optimize the joint angle q(t) so that the end effector reaches the desired position. At the same time, the loss function can also include other sub-loss functions such as various collision sub-loss functions, thus taking into account the potential collision risk of objects in the system.

[0219] For example, an exemplary code for this algorithm might look like this:

[0220]

[0221]

[0222] The advantages of this algorithm are as follows:

[0223] Simple: The prediction uses a first-order approximation of the speed, thus it is computationally efficient.

[0224] Fast computation: This algorithm does not require complex robot models and can quickly recalculate future states.

[0225] 2. Predicting joint angles using joint velocities and forward kinematics

[0226] In this algorithm, the joint velocity from the previous time step is used to predict the joint angles for the next few time steps. Correspondingly, as... Figure 5c As shown in this partial example of the disclosure, the method 100 for controlling the robotic arm may include:

[0227] S272. Determine the updated second current pose of the end effector in world coordinates. As mentioned above, based on the updated first joint angles, the information of the forward kinematics, and the first current pose, the pose of the end effector at time step t in world coordinates can be determined, which is the updated second current pose.

[0228] S274. Obtain the joint velocities of the robotic arm. Based on the second current pose, the updated second current pose, and the time step of the gradient descent algorithm, the joint velocities of the robotic arm, i.e., the joint velocities of the previous time step, can be obtained.

[0229] S276. Determine the predicted joint angles of the robotic arm. For example, the robot's configuration is determined by the joint angles q(t) and joint velocities v. q (t) describes the process, while the prediction of joint angles is based on velocity, for example, as shown in the following formula:

[0230] q(t+1)=q(t)+v q (t)Δt

[0231] Where q(t) is the joint angle at time step t; v q (t) represents the joint velocity at time step t. The time step size Δt can be equal to the time step size k of the gradient descent algorithm. t The time step Δt can also be equal to several time steps k of the gradient descent algorithm. t This refers to several time steps. Therefore, based on the joint velocity of the robotic arm in the previous time step t-1, the joint angle q(t) in the current time step t (i.e., the updated first joint angle) and the time step k of the gradient descent algorithm, the algorithm is determined. t This allows for the determination of predicted joint angles of the robotic arm.

[0232] S278. Determine the predicted pose of the end effector in world coordinates. As described above, based on the predicted joint angles of the robotic arm, the information of the forward kinematics, and the first current pose of the end effector, the predicted pose of the end effector in world coordinates can be determined.

[0233] like Figure 5d As shown, determining the loss function S130 may include:

[0234] S280. Determine the fifth sub-loss function and the fifth weight value of the corresponding fifth sub-loss function for the difference between the predicted pose and the updated second current pose.

[0235] By including a fifth sub-loss function and its weights in the loss function, the joint angle q(t) can be optimized by predicting the next joint configuration and minimizing the loss function. At the same time, the loss function can also include errors in the position and control force of the end effector and other sub-loss functions such as various collision sub-loss functions, thus taking into account the risk of potential collisions between objects in the system.

[0236] For example, an exemplary code for this algorithm might look like this:

[0237]

[0238]

[0239] The advantages of this algorithm are as follows:

[0240] Direct control: This algorithm directly controls the robot's joint angles, which are usually the inputs to the robot's actuators.

[0241] Versatility: This algorithm can be used for both position control and force control tasks.

[0242] Based on the principles of MPC, the prediction algorithms provided above offer a robust framework for controlling digital robots, especially in dynamic and constrained environments. By combining predictions of end effector position or joint angles, the system can predict future states and optimize control inputs accordingly. Predicting end effector position offers a simpler and more computationally efficient method, but is limited by the assumption of constant velocity; predicting joint angles provides more functionality and can directly control the robot's actuators, but requires careful management of velocity estimation errors.

[0243] A sphere representing the surface contour of an object

[0244] The inventors discovered that while using simplified geometric primitives (such as spheres) to represent 3D surfaces has been applied in applications requiring collision detection, conventional techniques rely on uniform sampling or simple geometric approximations. These techniques lack adaptability to local surface features that represent the geometric complexity of surface contours, such as curvature and vertex density. This can lead to poor accuracy or excessive computational cost when modeling complex geometries.

[0245] The inventors discovered that by using a predetermined number of spheres generated based on a 3D model to represent the surface contours of different regions of a target object, and by assigning a density of spheres in different regions to correspond to the geometric complexity of the surface contours in those regions, accurate modeling of complex geometries can be achieved while simultaneously balancing computational overhead. Correspondingly, such as Figure 6As shown in the example, this disclosure provides a method 400 for using spheres to represent the surface contours of different regions of a target object, comprising: step 410, obtaining a three-dimensional model of the target object; step 420, generating a preset number of spheres based on the three-dimensional model to represent the surface contours of different regions of the target object, wherein the density of spheres in the different regions corresponds to the geometric complexity of the surface contours of the different regions.

[0246] For example, different regions include a first region and a second region, where the geometric complexity of the surface contour of the first region is greater than that of the surface contour of the second region; the density of spheres in different regions corresponds to the geometric complexity of the surface contour of different regions, including: the density of spheres in the first region is greater than that in the second region.

[0247] Optionally, to determine the density weights of candidate sampling points in different regions of the target object, the density weights represent information about the geometric complexity of the surface contours of different regions, and can be obtained by:

[0248] a) Determine the neighborhood of the candidate sampling point based on the location of the candidate sampling point and the preset neighborhood size value.

[0249] b) Determine the density weight of the candidate sampling point based on the average distance between the candidate sampling point and other candidate sampling points in its neighborhood. Specifically, the density weight of the candidate sampling point can be determined according to the following formula:

[0250]

[0251] Among them, w i Density weights; ∈ represents the average distance; ∈ is a constant.

[0252] Specifically, ∈ is a small constant, its function being to prevent the denominator of the above formula from being 0, which would cause w to... i The value of w is unstable. As can be seen from the above formula, w i along with It increases as it decreases, because This represents the average distance between a candidate sampling point and other candidate sampling points in its neighborhood. The density of the candidate sampling points in the neighborhood increases, so the above formula assigns a higher density weight to the region of the candidate sampling points, ensuring that when determining sampling points from the candidate sampling points, priority is given to determining sampling points in regions with high geometric complexity of the surface profile.

[0253] The average distance can be determined using the following formula.

[0254]

[0255] in, The average distance, Let p be the set of other vertices in the neighborhood. i p is a candidate sampling point to serve as the neighborhood center. j Other alternative sampling points within the neighborhood.

[0256] Optionally, the 3D model is a polygonal mesh model, which consists of a set of vertices, edges and faces, and the alternative sampling points are the vertices of the polygonal mesh model.

[0257] Optionally, the neighborhood of the candidate sampling point can be determined by: taking the candidate sampling point as the center, defining the range of polygons whose number of edges between the candidate sampling point and the candidate sampling point is no greater than n as the neighborhood of the candidate sampling point, where the preset neighborhood size is n, and n is a positive integer.

[0258] Computer Graphics

[0259] Computer graphics is the field of generating photorealistic two-dimensional images using computer-based image rendering techniques. In this field, three-dimensional objects are represented using 3D object models, typically composed of point cloud triangular meshes, represented by the coordinates of vertices and faces, along with a color texture map of the model. Virtual cameras and lighting models are used to mathematically render the 3D objects into corresponding 2D images. Rendering methods themselves range from computationally inexpensive (e.g., rasterization) to computationally expensive (e.g., ray tracing). In the examples presented in this disclosure, rasterization-based image rendering techniques may be employed.

[0260] Specifically, in computer graphics, a polygonal mesh model defines the shape and outline of each 3D object. A polygonal mesh is constructed from smaller, interconnected planes (typically triangles or rectangles), resembling a 3D mosaic. Figure 1 They fit together perfectly. Each vertex in the polygonal mesh stores x, y, and z coordinate information. Then, each face of the polygon contains surface information.

[0261] Neighborhood

[0262] A neighborhood with a preset size of n can be called an n-ring neighborhood. This n-ring neighborhood defines the region around a vertex based on the connectivity of adjacent faces. For example... Figure 7 As shown, vertex p i The n-ring neighborhood includes the area from vertex p i Starting from the grid edge, consider all vertices reachable within n traversal steps, for example, a 1-ring neighborhood consisting of vertices p. iDirectly connected vertices (sharing an edge) form a 2-ring neighborhood, which is formed by vertices p. i It consists of all vertices of polygons whose number of sides is no greater than 2.

[0263] The parameter n, which controls the size of the n-ring neighborhood, is configurable. Larger n values ​​expand the neighborhood, capturing a wider range of surface features, while smaller n values ​​shrink the neighborhood, focusing on local details.

[0264] The adjacency relationships of an n-ring neighborhood can be calculated using a sparse adjacency matrix A, as shown in the following formula:

[0265]

[0266] The expansion of the neighborhood is achieved by iteratively multiplying by the adjacency matrix, as shown in the following formula:

[0267]

[0268] in, Let A be a vertex in the neighborhood of the k-ring. k Let δ represent the k-th power of matrix A. i Let be the vector representing the initial vertex.

[0269] By changing the parameter n, which controls the size of the n-ring neighborhood, the analysis range can be set to fit local or global geometry. Smaller ring neighborhoods limit computation to the nearest neighborhood and preserve finer details, while larger ring neighborhoods encompass a wider surface environment and can capture the global structure at the cost of higher computational cost, thus balancing detail preservation and computational efficiency.

[0270] In the examples disclosed herein, the three-dimensional model is a voxel model.

[0271] The voxel model consists of a set of voxels, and the candidate sampling point is the center of the outermost voxel of the voxel model.

[0272] Determining the neighborhood of candidate sampling points can include:

[0273] Centered on the candidate sampling point, the range of distances no greater than r between the candidate sampling point and the candidate sampling point is defined as the neighborhood of the candidate sampling point.

[0274] The preset neighborhood size is r, where r is a positive number.

[0275] Specifically, the voxel model simulates the surface geometry of a target object based on voxels, discretizing the 3D object into a structured mesh of cubic units (voxels). The resolution of the voxel model is controlled by the `voxel_resolution` parameter, which determines the number of subdivisions along the longest axis of the bounding box. Higher resolution provides finer sampling, but at the cost of increased computational complexity.

[0276] In some examples, a voxel model can be generated using a polygonal mesh model. For instance, the polygonal mesh model can be converted to a voxel model before sampling. Given that voxels in a voxel model are typically very dense, in some embodiments, only a subset of surface voxels (e.g., the outermost voxels) may be used for further sampling. In some embodiments, when the target object, whose surface contour is represented by a sphere, is a flexible object (e.g., a human hand), sampling can be performed using the vertex indices of the polygonal mesh model, without the conversion from the polygonal mesh model to a voxel model.

[0277] In some examples, generating a preset number of balls may include:

[0278] - Based on the density weight value, determine a preset number of sampling points from all candidate sampling points.

[0279] - Generate a preset number of balls with a defined sampling point as the center.

[0280] For example, determining a preset number of sampling points from all candidate sampling points can include:

[0281] a) Select one of the candidate sampling points as the initial sampling point.

[0282] b) Based on the Euclidean distance between each remaining candidate sampling point and the previous candidate sampling point, and the density weight of each remaining candidate sampling point, determine the next sampling point from the remaining candidate sampling points, until the predetermined number of sampling points are determined.

[0283] The remaining candidate sampling points are those that have not yet been identified as sampling points.

[0284] Density-weighted farthest point sampling

[0285] In an optional implementation, step b above may include: determining the comprehensive distance between each remaining candidate sampling point and the previous candidate sampling point based on the Euclidean distance between each remaining candidate sampling point and the previous candidate sampling point, and the density weight of each remaining candidate sampling point; and determining the remaining candidate sampling point with the largest corresponding comprehensive distance value as the next sampling point.

[0286] For example, the weighted distance can be determined using the following formula:

[0287]

[0288] Wherein, d′(p i ,p i Alternative sampling point p i Compared with the previously determined sampling point p j The combined distance between them, d(p) i ,p j p is the candidate sampling point. i Compared with the previously determined sampling point p j The Euclidean distance between them. For alternative sampling point p i Density weights.

[0289] j next =max i (d′(p i ,p j ))

[0290] The determined or selected next sampling point j next For the maximum composite distance value d′(p) i ,p j sampling point p) i .

[0291] The aforementioned formula is the formula for the DWFPS (Density-Weighted Farthest Point Sampling) strategy. The DWFPS strategy iteratively selects the vertex farthest from the previously determined center of the sphere, while incorporating density weights to favor regions with higher vertex density (i.e., regions with more complex geometry) in the selection of the next center. The DWFPS strategy ensures that sampling is more frequent in regions with high vertex density and less frequent in regions with low vertex density. Therefore, this method achieves adaptive sampling, effectively capturing both fine details and large-scale structures without redundant computation. The DWFPS strategy ensures both uniform spatial coverage and prioritizes regions with higher complexity.

[0292] Unlike the SDF (Signed Distance Function) method, which consumes a large amount of memory because it generates a surface distance field function suitable for any voxel in the 3D space of interest, the sphere-based computation of this disclosure is performed in pairs between a small number of spheres represented as a 3D point cloud. Consequently, the method of this disclosure consumes very little CPU / GPU memory, but is sufficient to generate a cost / reward function that can significantly alter robot motion.

[0293] Adaptive sphere radius

[0294] The size of the spheres can be adjusted according to changes in the surface contour of the target object. The method also includes: determining the sphere radii of a preset number of spheres based on a 3D model, and the sphere radii in different regions corresponding to the geometric complexity of the surface contour in those regions.

[0295] Optionally, the geometric complexity of the surface contours of the spheres in different regions corresponding to the sphere radii in different regions includes: the sphere radius in the first region is smaller than the sphere radius in the second region. Sampling points may include a first sampling point and a second sampling point.

[0296] Optionally, determining the sphere radius of a preset number of spheres may include: determining the sphere radius of each sphere based on the value of the preset radius and the value of the density weight of the sampling point corresponding to each sphere. Specifically, if the density weight of the first sampling point is greater than the density weight of the second sampling point, the sphere radius of the sphere corresponding to the first sampling point is less than the sphere radius of the sphere corresponding to the second sampling point.

[0297] Optionally, the density weight values ​​of the sampling points corresponding to each sphere can be normalized to obtain the normalized density weight values ​​of the sampling points corresponding to each sphere. The normalized density weight values ​​are positive numbers less than 1.

[0298] Specifically, to ensure that the density weights remain bounded, the density weights can be normalized using the following formula:

[0299]

[0300] Among them, w i For density weights, For normalized density weights.

[0301] Through the above normalization process, the density weight values ​​can be mapped to a range between 0 and 1, preventing extreme changes in the density weight values ​​and ensuring the stability of the density weight values.

[0302] Optionally, in order to normalize the density weight values ​​of the sampling points corresponding to each sphere, the method may further include: determining the sphere radius value of each sphere based on the preset radius value and the normalized density weight values ​​of the sampling points corresponding to each sphere.

[0303] For example, the radius of each sphere is determined based on the value of the preset radius, the value of the normalized density weight of the sampling points corresponding to each sphere, and the value of the scaling factor.

[0304] The scaling factor is used to control the sensitivity of the radius of each sphere to changes in the normalized density weight of the sampling point corresponding to each sphere.

[0305] Specifically, the radius of the sphere can be determined using the following formula:

[0306]

[0307] Where r0 is a preset radius, which can be defined by the user. s represents the normalized density weights, and s is the scaling factor.

[0308] The scaling factor *s* controls the sensitivity of the sphere's radius to changes in the density weight. The exponential term *e*... x Ensure the sphere radius r i The sphere can change smoothly and continuously without abrupt jumps, allowing it to seamlessly adapt to local geometric changes. In some embodiments, the scaling factor s can be set to 10, and the result can be visually checked by overlaying the generated sphere onto a polygonal mesh model of the target object. If the sphere is too small, a high scaling factor s can be used. If the sphere is too large, the scaling factor s can be decreased. In some embodiments, the scaling factor s can be 0.1, 1, 10, 100, 1000, or any value between 0.1 and 1000.

[0309] Using the above formula, the sphere used to represent the surface contour of different regions of the target object can shrink in areas where the candidate sampling points are densely distributed, and expand in areas where the candidate sampling points are sparsely distributed, thereby achieving adaptive coverage without introducing unnecessary computational costs.

[0310] Voxel-based sampling

[0311] In some embodiments, the 3D model can be, for example, a voxel model. When the 3D model is a voxel model, the density weights can be normalized according to the formula described above, and then the sphere radius can be determined based on the normalized density weights and the formula. Because the sphere radius scales according to the local voxel density, this allows for finer details in high-density regions and a larger sphere in sparse regions. When uniformity is prioritized over adaptability, a fixed sphere radius can be used instead of determining the sphere radius using the aforementioned method.

[0312] Using a voxel model as a 3D model has several advantages over using a polygonal mesh model in the following ways:

[0313] 1. Works independently of mesh topology, allowing spheres to be placed outside the mesh to capture volumetric features and discontinuous regions.

[0314] 2. Provides uniform coverage, making it effective for sparse or low-resolution grids.

[0315] Optionally, a preset number of sphere descriptor information can be generated. The descriptor information includes at least one of index information, position information, normal information, and sphere radius. The index information is an index list, which is a list of candidate sampling point indices {1,2,3,…,n}. IH A subset of}, n IH This represents the total number of spheres that indicate the surface contour of the target object. This descriptor information can be seamlessly integrated with tasks such as collision detection.

[0316] Optionally, the method for generating a preset number of spheres to represent the surface contours of different regions of the target object also includes: 1) acquiring the position information of all candidate sampling points in real time, and 2) determining the position information of all sampling points based on the position information and index information of all candidate sampling points. For example, once a specific set of hand mesh vertex indices I is selected... H (i.e., the complete hand mesh vertex index {1,2,3,…,n) IH Integer index i within the subset H List, n IH Using the total number of vertices in the hand mesh as the mounting points for the sphere on the hand surface (this set of indices only needs to be generated once and can be reused continuously), the position of the sphere on the hand surface can be calculated continuously in real time. The complete hand mesh vertices generated by the simpleHand model in the world coordinate system are represented as V. hand Let S represent the center position of the sphere on the hand surface in the world coordinate system. hand Then, S can be calculated very easily at any time using array index selection operations. hand =V hand [I H ].

[0317] Generate surface spheres to represent flexible objects

[0318] Based on the surface sphere representations generated for robot components, rigid and flexible objects (such as a human hand) in the world coordinate system, these representations can be directly used for different types of real-time robot control operations. For a flexible object (such as a human hand), the following steps can be taken to obtain a 3D model of the target object.

[0319] a. Obtaining real-time video stream data. For example, by acquiring a real-time video stream through a camera, a two-dimensional image of the target object can be contained within the video stream, thus allowing for the real-time acquisition of the two-dimensional image of the target object.

[0320] b. Generate a three-dimensional model of the target object based on the two-dimensional image of the target object.

[0321] Those skilled in the art will understand that there are different ways to generate a three-dimensional model of a target object based on a two-dimensional image of the target object. For example, in some embodiments, the following method steps can be used to generate a three-dimensional model.

[0322] c. Determine the bounding box of the target object in the 2D image. The bounding box of the target object in the 2D image can be determined based on an open-vocabulary object detection model and the input information corresponding to the target object. For example, the 2D image can be input into an open-vocabulary (e.g., "hand") object detection model. Using the corresponding vocabulary, the target object can be identified in the 2D image, and its position and size can be determined. Based on its position and size, a bounding box is generated in the 2D image to enclose the target object within the bounding box. The bounding box indicates the position and size of the target object in the 2D image.

[0323] d. Generate a 3D model of the target object based on the 2D image and the bounding box of the target object.

[0324] Taking the target object as an example, step d may include:

[0325] 1) Determine the 2D hand joints and 3D hand vertex positions of the target object.

[0326] 2) Based on the 2D hand joints and the 3D hand vertex positions, a three-dimensional model is obtained.

[0327] like Figure 8 An example of constructing a 3D model of a hand in real time from a camera image is shown, where the 3D model can be, for example, a polygonal mesh model.

[0328] Step 710: Input image. For example, an RGB image captured by a camera can be passed to an open-vocabulary object detection model, such as YOLO-World or GroundingDINO. The open-vocabulary object detection model YOLO-World is described in detail in the paper “Cheng, Tianheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan.Yolo-world: Real-time open-vocabulary object detection.” In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp.16901-16911.2024”, and all of its contents are incorporated herein by reference. The literature “Ren, Tianhe, Qing Jiang, Shilong Liu, Zhaoyang Zeng, Wenlong Liu, Han Gao, Hongjie Huang et al. Grounding DINO 1.5: Advance the “Edge” of Open-Set Object Detection. arXiv preprint arXiv:2405.10300(2024)” details the open-vocabulary object detection model GroundingDINO, and all of its contents are incorporated herein by reference.

[0329] Step 720, Object Detection. For example, an open-vocabulary object detection model can detect the location and size of possible hands, i.e., the bounding box of the hand. Optionally, the open-vocabulary object detection model can use the keyword "hand" as input to locate and detect hands in the input image. These open-vocabulary object detection models can detect various object categories, including hands, without retraining.

[0330] Step 730, Joint and Vertex Estimation. For example, the simpleHand model (consisting of a marker generator and a mesh regressor) can be used to process images with bounding box markers to estimate the 2D hand joints and 3D hand vertex positions, respectively. The simpleHand model is described in detail in the literature “Zhishan Zhou, Shihao Zhou, Zhi Lv, Minqiang Zou, Yao Tang, and Jiajun Liang. A simple baseline for efficient hand mesh reconstruction. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1367–1376, June 2024”, and its entire contents are incorporated herein by reference. Furthermore, the local camera coordinates of the hand vertices can be converted into 3D world coordinates along with depth information.

[0331] Step 740: Mesh Reconstruction. For example, the hand mesh can be created based on the hand vertices obtained in Step 730 and the triangular mesh generated from the hand model (e.g., the MANO model). The MANO model is described in detail in the literature "Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: modeling and capturing hands and bodies together. ACM Transactions on Graphics, pp. 1–17, 2017", and all of its contents are incorporated herein by reference.

[0332] The human hand is a highly flexible and deformable structure with multiple joints and degrees of freedom. Unlike rigid objects, its geometry constantly changes due to the bending, stretching, and rotational movements of the fingers and palm. Therefore, hard-coding the position of the sphere representing the hand's surface contour in a stationary coordinate system is impractical because it cannot accurately capture pose-related deformations. To address this issue, the previously mentioned adaptive surface sphere representation can be used to dynamically determine the vertex indices of the sphere placed on the simpleHand mesh template to represent the hand's surface contour. This method ensures that the sphere representing the hand's surface contour adapts to local geometric changes and deformations in real time.

[0333] Anti-collision loss function

[0334] The inventors further discovered that, in order to efficiently calculate the potential collision risk between the robot and objects in its surrounding environment, the robot links and object geometry can be represented as a set of small spheres, each defined by its center and radius, using the principle described above. This method decomposes complex shapes into simpler components, allowing for direct distance calculation and collision detection.

[0335] Specifically, each link of the robot and each object in the environment is approximated by a set of spheres. Let... Denotes the set of spheres that approximate the geometry of the robot, where It is the center of the i-th sphere associated with the robot. Let be the radius of the i-th sphere. Similarly, let be... Denotes the set of spheres for each object, where It is the center of the j-th sphere associated with the robot, r j obj It is the radius of the i-th sphere.

[0336] Using these spheres simplifies the process of identifying potential collisions between robots and objects. The positions of these spheres are adjusted in real time based on the robot's movement, and collisions can be continuously and efficiently checked by calculating the distances between all robot-object sphere pairs.

[0337] Figure 9 The images show some examples of collision sphere representations. The top image shows the original mesh of the object, and the bottom image shows the representations of the corresponding small spheres 510, 520, and 530.

[0338] Robot-object collision loss function

[0339] By utilizing the spherical representations of the robot and the object, along with their respective poses, robot-object collision losses can be calculated, thus achieving effective collision avoidance. This method simplifies the complex geometry of robot links and objects into manageable spherical regions, enabling efficient calculation of pairwise distances between all robot-object spherical combinations.

[0340] To effectively handle collision avoidance, the geometry of the robot links and objects is first represented as a set of small spheres in each local coordinate system, defined by their center position and radius. Let... and let Let represent the sets of collision spheres for the robot and the object, respectively. and It is the center of the i-th robot sphere and the j-th object sphere. and This represents the corresponding radius. N is the number of robot links, M is the number of objects in the scene, and B is the number of balls for each link or object.

[0341] Because the poses of the robot and objects change dynamically, the center of each sphere must be transformed from its local coordinate system to the robot's reference (world) coordinate system to reflect the current configuration of the robot and objects. This transformation is achieved using the robot's forward kinematics and the pose matrix of each object. Let FK n (θ′) represents the positive kinematic transformation matrix of the nth robot link under a given joint configuration θ′, and The 6D pose transformation matrix for the target object from its stationary coordinate system to the robot's stationary coordinate system, with the center of the transformable sphere as follows:

[0342]

[0343] After transforming the centers of all spheres to the global coordinate system, calculate the effective pairwise distance D between each robot-object sphere pair (i,j). ij As shown below:

[0344]

[0345] This distance D ij This represents the distance between the surfaces of two spheres.

[0346] In one alternative implementation, the robot-object collision loss L can be... robot-obj As a sub-loss function within the loss function.

[0347] Specifically, to define the cost of penalizing smooth collisions involving close proximity and overlap, a cosine-based penalty function f can be used. cos (D ij When D ij When the distance exceeds the threshold `skin_dist`, the function smoothly decays to zero. The penalty function is defined as follows:

[0348]

[0349] Robot-Object Collision Loss L robo_obj Calculate the average collision cost for all robot-object sphere pairs:

[0350]

[0351] This loss function represents the overall collision risk in the workspace. L is minimized through trajectory optimization. robot-obj It can effectively encourage robots to avoid potential collision areas, thereby promoting safe and efficient movement in environments with multiple obstacles.

[0352] Robot-human collision loss function

[0353] The human body is a flexible object. As mentioned above, combined with human pose estimation algorithms, the human body can also be represented by a set of spheres, similar to object modeling. These spheres can then be transformed based on the estimated human pose to accurately reflect the body's spatial configuration. Using this representation, a collision loss function between the robot and flexible objects like the human body can be calculated in a manner similar to object collision loss. This allows the system to account for potential interactions or collisions with humans, thereby enhancing safety and adaptability in dynamic environments.

[0354] Robot self-collision loss function

[0355] In one alternative implementation, the robot's self-collision loss L can be... self-collision As a sub-loss function within the loss function.

[0356] Specifically, to calculate the robot's self-collision loss, a method similar to that used for robot-object collision loss can be employed, but only the robot's internal structure is considered. Each link of the robot is represented by a set of small spheres, each with a center position and radius. Let... Let represent the set of collision spheres across all links. Since the robot's configuration changes dynamically, the positive kinematic transformation matrix FK for each link i is used. n (θ′) transforms the center of each sphere from its local link coordinate system to the global (robot reference) coordinate system. Then, the transformation center of each sphere is determined by... Provided.

[0357] After transforming the centers of all spheres to the global coordinate system, calculate the effective distance D between each pair of (i,j) spheres belonging to different links. ij ,Right now To obtain a smooth self-collision cost, the same cosine-based penalty function f can be used. cos (D ij When D ij When the distance exceeds the threshold skin_dist, the function will smoothly decay to zero.

[0358] Finally, the total self-collision loss L self-collision The calculation is the average of the self-collision costs of all pairs on all links, excluding pairs within the same link:

[0359]

[0360] Minimize L through trajectory optimization self-collision This can encourage robots to adopt joint configurations that avoid self-collision, promoting safer movement and internal coordination across links.

[0361] Robot-physical surface collision loss function

[0362] In one alternative implementation, the collision loss function L between the robot and the physical surface can be... robot-surface As a sub-loss function within the loss function.

[0363] Specifically, in addition to robot-object collision avoidance and robot-on-collision avoidance, robots also need to avoid collisions with certain physical surfaces in the environment, such as tables, floors, or walls. These surfaces can be represented as a set of planes in 3D space, each defined by a normal vector and an offset distance from the origin. Let each physical surface S... k By its normal vector and offset d k This means that the plane equation for each surface is... in It is any point on the plane.

[0364] To calculate the collision risk between the robot and a surface, the shortest distance between each colliding ball on the robot and the surface in the environment is measured. For each sphere... in It is the center of the collision sphere in the global coordinate system, D ik To surface S k The distance is calculated as follows:

[0365]

[0366] This effective distance D ik Represents the relationship between the surface of a sphere and the physical surface S k The space between them; to obtain a smooth self-collision cost, the same cosine-based penalty function f is used. cos (D ik When D ik When the distance exceeds the threshold skin_dis, the function smoothly decays to zero.

[0367] Then, the total collision loss L between the robot and the physical surface is calculated. robot-surface The calculation is the average cost of these individual collisions across all robot spheres and associated physical surfaces:

[0368]

[0369] In the examples disclosed herein, the loss function Loss can be as follows:

[0370] Loss=λλ1L dist +λ2L dist +λ3L n_dor +λ4L robot-obj +λ5L self-collision

[0371] +λ6L robot-surface +λ7L angle-limit +λ8L speed-limit

[0372] Each term in the loss function penalizes deviations from the desired pose, orientation, and spatial constraints, including collision risks with objects, surfaces, and the robot itself, as well as joint limitations. In the examples provided in this disclosure, if the object is a flexible object such as a human body, the loss function Loss may further include, as described above, a sub-collision loss function for the robot and flexible objects such as a human body, along with their corresponding weights.

[0373] Using this loss function, joint angles θ can be updated via gradient descent to generate the robot trajectory. This process continues to produce updated θ until each loss term is below a predefined tolerance, indicating that the robot's current pose closely matches the desired grasping pose.

[0374] In some examples of this disclosure, when the value of each sub-loss function in the loss function is below a predetermined threshold, the robot can receive a first instruction that instructs the end effector to perform a grasping operation. For example, the first instruction could be a CLOSE() command, which is sent to the gripper to close and grasp the object. This method provides a robust and efficient approach for achieving collision-free, precise grasping in dynamic environments.

[0375] The robot can also receive other instructions. For example, when the robot's task is to grasp a target object, a second instruction can be sent to the end effector, such as the CLOSE() command, to close the end effector and grasp the target object. This method provides a robust and efficient way to accurately grasp target objects in dynamic environments. For example, the third instruction can be the GOTO() command, and the fourth instruction can be the OPEN() command. Combining the GOTO() and CLOSE() commands can achieve efficient and automatic object grasping. For automatic placement, the GOTO() command can also be used to move the end effector to the target pose, and then the OPEN() command can be used to release the object.

[0376] Figure 10 This illustrates the process of a robot's end effector gripping a target object. The white ghosting is a digital rendering of the robot and object using pose estimation results in real time. Figure 10 (a) in Figure 10 In (d), the real robot's gripper gradually approaches the target grasping pose. Figure 10 (e) shows the gripper executing the close command and grabbing the teapot. Figure 10(f) shows the robotic arm lifting the teapot, indicating a successful grab.

[0377] like Figure 11 As shown in the example of this disclosure, determining the loss function S130 further includes:

[0378] S820. Obtain the state information of the robotic arm. The state information may indicate one or more of the following: whether the robotic arm is holding an object, whether the robotic arm is moving, whether the robotic arm is stationary, whether the current pose of the robotic arm matches the target pose, whether the robotic arm is grasping an object, and whether the robotic arm is releasing the held object.

[0379] S830. Based on the state information of the robotic arm, adjust the weight values ​​corresponding to each sub-loss function of the loss function.

[0380] By adjusting the weight values ​​corresponding to each sub-loss function based on the robotic arm's status information, the number and types of sub-loss functions within the overall loss function can be adjusted. For example, the number and types of sub-loss functions can be adjusted by changing the corresponding weight values ​​based on the type of control command and whether there is a load in the end effector.

[0381] Optionally, in the examples of this disclosure, determining the loss function S130 may include:

[0382] S810. Based on the instructions received by the robotic arm and whether the end effector performs the grasping action, the status information of the robotic arm can be determined.

[0383] In the examples disclosed herein, the robot's state information can be determined based on the type of received command, proximity to the target object, and whether the end effector holds a payload. This is crucial for optimizing control and safety, as well as effectively managing the objectives defined by the loss function. The robot's state affects the priority of different constraints and losses in the objective function; that is, it affects the weight coefficients of various sub-objective functions within the objective function, enabling the robot to adjust its behavior according to the requirements of the current task. The criteria for determining the robot's state include whether it is moving while the end effector holds a payload and whether the end effector is grasping or releasing an object. Based on the type of underlying commands received by the robot and whether the end effector holds a payload, the robot's state can be defined into eight types: IDLE_WITHOUT_PAYLOAD (stationary with no payload), IDLE_WITH_PAYLOAD (stationary with payload), MOVING_WITH_PAYLOAD (moving with payload), MOVING_WITHOUT_PAYLOAD (moving without payload), PRE_GRASPING (pre-grabbing), GRASPING (grabbing), PRE_RELEASING (pre-releasing), and RELEASING (releasing). These eight states are managed through a state machine framework, which dynamically transitions between them based on the type of received commands and whether the end effector holds a payload. This state machine architecture allows for system transitions to ensure optimized task-specific operations based on the robot's current role and environment.

[0384] Figure 12 The diagram illustrates the transitions between the eight states described above. Both IDLE_WITHOUT_PAYLOAD and IDLE_WITH_PAYLOAD indicate that the robot is in a stationary mode, waiting for the next command. The difference between IDLE_WITHOUT_PAYLOAD and IDLE_WITH_PAYLOAD is that IDLE_WITHOUT_PAYLOAD indicates that there is no load in the end effector, while IDLE_WITH_PAYLOAD indicates that the end effector holds a payload. After the robot completes a task, its default state is either IDLE_WITHOUT_PAYLOAD or IDLE_WITH_PAYLOAD. If the robot successfully completes the grasping operation and the end effector holds a payload, it enters IDLE_WITH_PAYLOAD; otherwise, it enters IDLE_WITHOUT_PAYLOAD.

[0385] When the robot receives the GOTO() command, its state transitions to either MOVING_WITHOUT_PAYLOAD or MOVING_WITH_PAYLOAD. If the end effector holds a payload, it transitions to MOVING_WITH_PAYLOAD; otherwise, it enters MOVING_WITHOUT_PAYLOAD. In MOVING_WITH_PAYLOAD, additional collision constraints are activated to account for the payload's interaction with surrounding objects and surfaces, thus protecting both the robot and the payload. Conversely, in MOVING_WITHOUT_PAYLOAD, the system reduces the emphasis on collisions with external constraints and focuses instead on efficient movement to the next operating position.

[0386] When the end effector needs to grasp or release an object, the state machine transitions the robot's state from the pre-action state to the execution state. Specifically, when the end effector needs to grasp an object, the state machine transitions the robot's state from PRE_GRASPING to GRASPING; when the end effector needs to release an object, the state machine transitions the robot's state from PRE_RELEASING to RELEASING, thereby achieving accurate and safe object grasping.

[0387] When the robot is in PRE_GRASPING, it indicates that the robot is preparing to align its end effector and accurately approach the target object, thus optimizing its localization before performing grasping. After the end effector reaches the target position and receives the CLOSE() command, the robot transitions to GRASPING, a state in which the end effector can safely grasp the object. Similarly, when the robot is in PRE_RE LEASING, it indicates that the robot is preparing to accurately locate itself to place the object in the designated area. When the robot is ready to release the object held by the end effector, it receives the OPEN() command, at which point the state machine transitions the robot's state to RELEASEING, thus achieving accurate and safe object placement. When the robot is in PRE_GRASPING or PRE_RELEASING, the types, number, and weight coefficients of the sub-objective functions in the objective function are adjusted accordingly to minimize pose error.

[0388] Adjusting the types, number, and weighting coefficients of sub-objective functions within the objective function, based on the robot's state, can yield smooth, safe, and adaptive trajectories for the end effector, thereby supporting the efficient execution of complex tasks. This state-based configuration enables the robot to maintain a high level of reliability and adaptability in dynamic environments, as each state is paired with a set of optimization constraints that satisfy the current task's operational requirements.

[0389] Real-time pose estimation is crucial for achieving accurate and adaptive robotic manipulation, especially in dynamic environments where objects may move or change their spatial relationships relative to each other or the robot. For tasks such as grasping-placement, assembly, and binding, accurate estimation of the pose of individual objects and the interactions between objects and the robot is essential. This becomes particularly important when managing multiple objects or handling complex operations, such as attaching or detaching objects from a robot's end effector.

[0390] Figure 13 This illustrates the interaction between the robot and the object during a grasping-and-place operation. The white ghosting is a digital rendering of the robot and object in real time using pose estimation results. Figure 13 (a) in Figure 13 In (b), the gripper at the end of the real robot gradually approaches the handle of the teacup. Figure 13 In step (c), the gripper successfully grasps the teacup, attaching the object to the gripper. From Figure 13 (d) in Figure 13 In (e), the teacup's position is constantly updated based on its relative position to the grippers. Figure 13 In step (f), the gripper releases the teacup, separating it from the gripper.

[0391] Object pose estimation

[0392] Object pose estimation is the process of determining the position and orientation of an object in space, that is, determining the object's 6D pose. It is typically derived from image data and is achieved by estimating the pose of a target object relative to the camera that captured the image. Object pose estimation is a key technology in fields such as augmented reality, virtual reality, robotic manipulation, and hand-object interaction. Depending on the application requirements, the pose of the target object is estimated to different degrees of freedom (DoF), such as 3DoF including only 3D rotation, 6DoF including 3D translation, or 9DoF including 3D rotation, 3D translation, and also estimating the object's 3D size. Understanding the spatial arrangement of objects in the environment helps robots perform operations on target objects.

[0393] Real-time pose estimation systems can provide direct input to robot control systems, enabling them to perform precise actions. For example, during object grasping, pose data allows the control system to calculate optimal methods and maneuvering strategies to minimize the risk of collisions or slippage. Pose estimation can also go beyond standalone object recognition and visual tracking, incorporating feedback from the robot's own sensors. Signals such as the timing of connection or separation events, payload state, and joint positions are crucial for updating the pose of the object relative to the robot's end effector. This integration can significantly improve the accuracy of pose estimation because the object's pose is no longer estimated solely through external observation but is directly correlated with the robot's precise motion data, thus reducing errors caused by occlusion or visual noise. Furthermore, when an object is attached to the robot, the system can utilize more advanced maneuvering strategies, such as treating the combined object-robot structure as a single entity to avoid collisions and for path planning. Thus, in some examples of this disclosure, when the end effector holds an object, a sphere representing the surface profile of the end effector is stacked with a sphere representing the object, wherein the stacked spheres represent the object as an extension of the end effector to represent the surface profile of the extended end effector.

[0394] In more complex applications, such as when the robot needs to perform tasks involving multiple bound objects, it must maintain the relative stability of individual objects while managing the collective pose of the object group. Real-time data feedback from sensors mounted on the robotic arm enables the robot to detect and compensate for unexpected events, such as slippage, deformation, and external interactions; this real-time feedback data can include tactile and proprioceptive data. By combining visual pose tracking with feedback data from sensors mounted on the robotic arm, the robot's control strategy can be dynamically adjusted to ensure robustness and accuracy in object-robot interactions. This integration of multimodal data is crucial for improving the overall accuracy of operational tasks in complex and unpredictable environments.

[0395] In one alternative implementation, the planning method further includes: receiving image information of all objects and the robotic arm within the workspace of the robotic arm sent by the image acquisition module; and displaying the image information on the user interface of the display.

[0396] In one alternative implementation, the planning method further includes generating instructions, such as control commands, in response to user actions on the user interface. For example, the user interface may be a GUI (Graphical User Interface), on which users can conveniently and intuitively perform operations.

[0397] Graphical User Interface (GUI)

[0398] The graphical user interface (GUI) is the interface through which users can easily and intuitively control the robot. The GUI can display video from each camera and rendered results of the 3D scene. To control the robot, users simply select a command from the toolbar and then click on the interactive video panel.

[0399] Figure 14 A graphical user interface (GUI) 910 is displayed, allowing users to easily and intuitively control the robot. The GUI displays video from each camera and the rendered results of the 3D scene. To control the robot, the user simply selects a command from the GUI toolbar and clicks on the interactive video panel.

[0400] The GUI system is designed for intuitive and user-friendly operation, enabling seamless interaction with the real-time digital twin reconstruction of the robot and its environment. Users can visually interact with the reconstructed scene and issue high-level task commands directly within the GUI. For example, a user can left-click on an object to select it, move it to a new location, and choose to adjust its rotation before placement. After entering the required command, the user clicks the "Robot Start" button to execute the command. Users can enter multiple high-level task commands before starting. In the backend, the processor translates these high-level commands into a series of low-level commands, which serve as input to the trajectory planning method in this embodiment for the robot to execute; these low-level commands include GOTO(), OPEN(), CLOSE(), etc. The robot, whether physical or simulated, executes these commands, and its movement is dynamically reflected in the real-time digital twin reconstruction of the GUI. Furthermore, the GUI can record an event log, providing users with feedback on the success or failure of each high-level task, ensuring transparency and facilitating troubleshooting.

[0401] In one alternative implementation, the planning method further includes: receiving image information of all objects and the robotic arm within the workspace of the robotic arm sent by the image acquisition module; rendering the image information to obtain rendering information; sending the rendering information to a display for real-time display; and generating instructions in response to user operations on the user interface.

[0402] The inventors discovered that the initial implementation of current GUI visualization pipelines is inefficient, severely impacting the overall system's frames per second (FPS), including tasks such as mask segmentation, pose estimation, and control processing. The main bottleneck lies in inefficient data transfer between the CPU and GPU, with frequent memory copying leading to latency spikes and performance degradation. The inventors also found that the rendering pipeline operates in a synchronous blocking mode, preventing other processing threads from executing in parallel and causing further latency. Using an unoptimized GUI framework exacerbates these problems because it fails to utilize modern GPU acceleration techniques (such as direct GPU memory sharing), resulting in excessive CPU utilization and slower rendering times for scene meshes and surface spheres.

[0403] Using the spheres generated in some examples of this disclosure to represent the surface contours of target objects can optimize the rendering pipeline. For example, memory sharing can be achieved using the GPU integrated with PyTorch-OpenGL. This method achieves zero-copy data sharing between PyTorch tensors and OpenGL buffers, eliminating redundant memory transfers and enabling direct GPU rendering of visual output. The solutions provided in some embodiments of this disclosure can implement custom OpenGL shaders, thereby enabling direct processing of transformations and lighting effects on the GPU, reducing the computational burden on the CPU. The solutions provided in some embodiments of this disclosure can separate rendering tasks from data processing using parallel threads, ensuring that GUI updates no longer hinder critical system operations. In addition, the frame rate capping mechanism dynamically adjusts GUI updates according to system load to maintain FPS stability. Further optimizations include lazy loading and caching strategies for static meshes, thereby reducing redundant computation and memory usage. These enhancements collectively create a scalable, responsive, and real-time visualization system that can run seamlessly with other computationally intensive tasks.

[0404] Example 2

[0405] This embodiment provides a control system 1000 for a robotic arm, such as... Figure 15 As shown, the control system includes one or more processors 1020, which execute computer programs to control the robotic arm to implement the control method in Embodiment 1.

[0406] In some examples of this disclosure, the control system 1000 may further include an image acquisition module 1010. The image acquisition module is used to acquire image information of all objects and the robotic arm within its workspace. The image acquisition module 1010 can send the image information to the processor 1020 for processing.

[0407] For example, the image acquisition module 1010 can be implemented using a camera. The image acquisition module 1010 may also include a depth camera.

[0408] In some examples of this disclosure, the control system 1000 may further include a robotic arm 1030 and a sensor 1040. The robotic arm includes a base, at least one link, and a joint connecting the at least one link, wherein the at least one link includes an end effector. The sensor 1040 may be disposed on the end effector of the robotic arm; for example, the sensor 1040 is a tactile sensor.

[0409] The processor 1020 can also be used to: receive image information of all objects and the robotic arm within the workspace of the robotic arm 1030 sent by the image acquisition module 1010; acquire feedback information sent by the sensor 1040; and determine whether the end effector of the robotic arm 1030 holds an object based on the image information and the feedback information.

[0410] In some examples of this disclosure, the control system 1000 may further include a display 1050, which displays a user interface and displays image information processed by the processor 1020 in real time within the user interface. The user interface may also be a GUI.

[0411] In some examples of this disclosure, the display 1050 can also be used to: receive user operations via a user interface; the processor 1020 can also be used to generate control commands based on the user operations.

[0412] In one alternative implementation, an image acquisition module and / or a display are included. The image acquisition module is used to acquire image information of all objects and the robotic arm within the workspace; the display is used to show the user interface and display the received 3D rendering information in real time within the user interface.

[0413] In some embodiments, the system can incorporate a robust architecture through multithreading to manage real-time operation and interaction, ensuring seamless coordination between robot control, user input, and real-time digital twin visualization.

[0414] Figure 16 The multithreaded system architecture in some embodiments is shown. Each solid line represents a queue for a thread or process to send information.

[0415] The GUI thread serves as the main interface, receiving real-time SceneInfo from the Pose thread to display 3D models of the robot, objects, and their environment. It also handles all user interactions, allowing users to generate high-level commands, which are then sent to the Plan thread for processing.

[0416] In the robot communication process, the robot API continuously queries the robot hardware for its current θ and sends the required θ (actual-q) calculated by the control thread.

[0417] This process also monitors the status of the robot arm end effector, sending "off" or "on" signals as needed to perform gripping or releasing operations.

[0418] The planning thread is responsible for breaking down high-level user commands into a series of low-level commands and combining them with waypoint-based robotic arm movements for precise execution. For each low-level command, the planning thread dynamically creates a corresponding control thread and passes the robot's state to it for execution. The control thread uses advanced optimization techniques to calculate the required robot trajectory and execute the low-level tasks. After each execution, the control thread returns a result indicating the success or failure of the movement, which the planning thread uses to continue planning, updating the robot's state, and recording events.

[0419] The pose thread processes camera frames in real time to estimate the poses of the robot and target objects from 2D images. It also classifies and identifies objects, packages this data into SceneInfo for distribution to the GUI thread (GUI / mainthread) for real-time digital twin visualization, and distributes it to the planning and control threads for intelligent robot operation.

[0420] This multi-threaded architecture utilizes asynchronous, non-blocking queues for inter-thread communication, ensuring efficient and real-time data exchange. Threads are dynamically created and destroyed as needed, especially for tasks requiring the execution of low-level commands, optimized resource utilization, and enhanced system responsiveness. This design enables the system to provide real-time performance, robust task execution, and intuitive user interaction for dynamic and intelligent robotic operations.

[0421] For the system embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The system embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs.

[0422] Example 3

[0423] Figure 17 This is a schematic diagram of the structure of an electronic device according to an example embodiment of the present disclosure. The electronic device includes a memory, a processor, and a computer program stored in the memory and used to run on the processor. When the processor executes the computer program, it implements the control method of the robotic arm of any of the above embodiments. Figure 17 The illustrated electronic device 30 is merely an example; for instance, those skilled in the art will understand that a portion of the processor, such as an ASIC processor, may include memory. Figure 17 The electronic device 30 shown should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0424] like Figure 17 As shown, the electronic device 30 can be manifested as a general-purpose computing device, such as a server device. The components of the electronic device 30 may include, but are not limited to: at least one processor 31, at least one memory 32, and a bus 33 connecting different system components (including memory 52 and processor 31).

[0425] Bus 33 includes a data bus, an address bus, and a control bus.

[0426] The memory 32 may include volatile memory, such as random access memory (RAM) 321 and / or cache memory 322, and may further include read-only memory (ROM) 323.

[0427] The memory 32 may also include a program tool 325 (or utility) having a set (at least one) program module 324, such program module 324 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0428] The processor 31 executes various functional applications and data processing by running computer programs stored in the memory 32, such as the robotic arm control method provided in any of the above embodiments.

[0429] Electronic device 30 can also communicate with one or more external devices 34 (e.g., keyboard, pointing device, etc.). This communication can be performed through input / output (I / O) interface 35. Furthermore, electronic device 30 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public network, such as the Internet) via network adapter 36. As shown, network adapter 36 communicates with other modules of electronic device 30 via bus 33. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with electronic device 30, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0430] It should be noted that although several units / modules or sub-units / modules of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0431] Example 4

[0432] This disclosure also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the robotic arm control method provided in any of the above embodiments.

[0433] Example 5

[0434] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described control method for a robotic arm.

[0435] In some embodiments of this disclosure, the readable storage medium and computer program product may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0436] Example 6

[0437] This disclosure also provides a computer program that, when executed by one or more processors, can implement the above-described robotic arm control method.

[0438] The program code for executing the computer program product disclosed herein can be written in any combination of one or more programming languages. The program code can be executed entirely on a user device, partially on a user device, as a stand-alone software package, partially on a user device and partially on a remote device, or entirely on a remote device.

[0439] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of the present invention is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but all such changes and modifications fall within the scope of protection of the present invention.

Claims

1. A control method for a robotic arm, the robotic arm comprising a base, at least one link, and a joint connecting the at least one link, wherein, The at least one link includes an end effector, characterized in that the control method includes: Obtain a three-dimensional model of the target object, which includes the robotic arm and the obstacle; Based on the three-dimensional model, a preset number of spheres are generated to represent the surface contours of different regions of the target object, wherein the density of spheres in different regions corresponds to the geometric complexity of the surface contours of the different regions. Obtain the target pose of the end effector in world coordinates and the positive kinematics of the robotic arm; Based on the first joint angle of the robotic arm, the information of the forward kinematics, and the first current pose of the end effector, the second current pose of the end effector in world coordinates is determined, wherein the first current pose is the pose of the end effector in the coordinate system of the joint connected to the end effector; Determine a loss function, the loss function including a first loss function corresponding to the difference between the target pose and the second current pose; Based on the loss function, the first joint angle is updated using a gradient descent algorithm to control the movement of the robotic arm; Wherein, if the obstacle is a rigid object, the loss function further includes a sixth sub-loss function corresponding to avoiding collision between the robotic arm and the rigid object, the sixth sub-loss function corresponding to a sixth weight value; and, determining the loss function includes: Based on the ball of the robotic arm and the ball of the rigid object, all first ball pairs are determined, wherein the first ball pair consists of any one ball of the rigid object and any one ball of the robotic arm; The sixth sub-loss function is determined by comparing the spherical distance between the two balls in the first ball pair with a preset threshold. Wherein, if the obstacle is a flexible object, the loss function further includes a seventh sub-loss function corresponding to avoiding collision between the robotic arm and the flexible object, the seventh sub-loss function corresponding to a seventh weight value; and the determination loss function further includes: Based on the ball of the robotic arm and the ball of the flexible object, all second ball pairs are determined, wherein the second ball pair consists of any one ball of the robotic arm and any one ball of the flexible object; The seventh sub-loss function is determined by comparing the spherical distance between the two balls in the second ball pair with a preset threshold.

2. The control method for the robotic arm as described in claim 1, characterized in that, The target pose includes the coordinates of a first three-dimensional grasping point, a first grasping direction vector, and a first grasping direction normal vector; The second current pose includes the coordinates of the second three-dimensional grasping point, the second grasping direction vector, and the second grasping direction normal vector.

3. The control method for the robotic arm as described in claim 2, characterized in that, The first loss function includes: The first sub-loss function corresponds to the difference between the coordinates of the first three-dimensional grasp point and the coordinates of the second three-dimensional grasp point, and the first sub-loss function corresponds to the first weight value; The second sub-loss function corresponds to the difference between the first grasping direction vector and the second grasping direction vector, and the second sub-loss function corresponds to the second weight value; The third sub-loss function corresponds to the difference between the first grasping direction normal vector and the second grasping direction normal vector, and the third sub-loss function corresponds to the third weight value.

4. The control method for the robotic arm as described in any one of claims 1-3, characterized in that, The control method further includes: Based on the updated first joint angle, the information of the positive kinematics, and the first current pose, the updated second current pose of the end effector in world coordinates is determined; Based on the second current pose, the updated second current pose, and the time step of the gradient descent algorithm, the first velocity of the end effector is obtained; Based on the first velocity of the end effector, the updated second current pose, and the time step of the gradient descent algorithm, the predicted pose of the end effector in world coordinates is determined. The loss function further includes a fourth sub-loss function corresponding to the difference between the predicted pose and the updated second current pose, and the fourth sub-loss function corresponds to a fourth weight value.

5. The control method for the robotic arm as described in any one of claims 1-3, characterized in that, The control method further includes: Based on the updated first joint angle, the information of the positive kinematics, and the first current pose, the updated second current pose of the end effector in world coordinates is determined; Based on the second current pose, the updated second current pose, and the time step of the gradient descent algorithm, the speed of the joints of the robotic arm is obtained; Based on the speed of the robotic arm's joints, the updated first joint angle, and the time step of the gradient descent algorithm, the predicted joint angle of the robotic arm is determined. Based on the predicted joint angles of the robotic arm, the information of the forward kinematics, and the first current pose of the end effector, the predicted pose of the end effector in world coordinates is determined. The loss function further includes a fifth sub-loss function corresponding to the difference between the predicted pose and the updated second current pose, and the fifth sub-loss function corresponds to a fifth weight value.

6. The control method for the robotic arm as described in any one of claims 1-3, characterized in that, If the obstacle is a flexible object, obtaining the three-dimensional model of the target object includes: Obtain data from the real-time video stream; A three-dimensional model of the flexible object is generated based on the two-dimensional image of the flexible object in the video stream.

7. The control method for the robotic arm as described in any one of claims 1-3, characterized in that, The loss function also includes an eighth sub-loss function that corresponds to avoiding collisions between the links of the robotic arm, and the eighth sub-loss function corresponds to an eighth weight value; The determination of the loss function also includes: Based on the balls of the robotic arm, determine all third ball pairs, wherein the third ball pair consists of any two balls on different links of the robotic arm; The eighth sub-loss function is determined by comparing the spherical distance between the two balls in the third ball pair with a preset threshold.

8. The control method for the robotic arm as described in any one of claims 1-3, characterized in that, The target object also includes the physical surface within the workspace of the robotic arm, and the loss function also includes a ninth sub-loss function corresponding to avoiding collisions between the robotic arm and the physical surface, the ninth sub-loss function corresponding to a ninth weight value; The determination of the loss function also includes: Based on the ball in the robotic arm and the ball on the physical surface, all fourth ball pairs are determined, wherein each fourth ball pair consists of any one ball in the robotic arm and any one ball on the physical surface. The ninth sub-loss function is determined by comparing the spherical distance between the two balls in the fourth ball pair with a preset threshold.

9. The control method for the robotic arm as described in any one of claims 1-3, characterized in that, The loss function also includes a tenth sub-loss function for limiting the angle range of the joint angles of the robotic arm, and the tenth sub-loss function corresponds to the tenth weight value; The determination of the loss function also includes: The tenth sub-loss function is determined by comparing the first joint angle of the robotic arm with a preset joint angle threshold.

10. The control method for the robotic arm as described in any one of claims 1-3, characterized in that, The loss function also includes an eleventh sub-loss function for limiting the speed of the end effector, the eleventh sub-loss function corresponding to an eleventh weight value; The determination of the loss function also includes: Based on the updated first joint angle, the information of the positive kinematics, and the first current pose, the updated second current pose of the end effector in world coordinates is determined; The second velocity of the end effector is obtained based on the second current pose, the updated second current pose, and the time step of the gradient descent algorithm. The eleventh sub-loss function is determined based on the comparison between the second speed of the end effector and a preset speed threshold, wherein the preset speed threshold includes a maximum speed threshold.

11. The control method for the robotic arm as described in claim 10, characterized in that, The preset speed threshold also includes a minimum speed threshold.

12. The control method for the robotic arm as described in claim 3, characterized in that, When the value of each sub-loss function of the loss function is lower than a predetermined threshold, the current pose of the robotic arm matches the target pose.

13. The control method for the robotic arm as described in claim 12, characterized in that, The control method further includes: When the value of each sub-loss function of the loss function is lower than a predetermined threshold, a first instruction is received, which instructs the end effector to perform an object grasping operation.

14. The control method for the robotic arm as described in claim 3, characterized in that, The determination of the loss function also includes: Obtain the state information of the robotic arm, wherein the state information indicates at least one of the following: Does the robotic arm hold an object? Is the robotic arm moving? Is the robotic arm stationary? Does the current pose of the robotic arm match the target pose? Is the robotic arm currently grasping an object? Is the robotic arm releasing the object it is holding? Based on the state information, adjust the weight values ​​corresponding to each sub-loss function of the loss function.

15. The control method for the robotic arm as described in claim 14, wherein determining the loss function further includes: The status information is determined based on the control commands received by the robotic arm and whether the end effector performs the action of grasping an object.

16. The control method for the robotic arm as described in claim 15, characterized in that, The control method further includes: Once the end effector holds an object, a sphere representing the surface profile of the end effector is stacked with a sphere representing the object, wherein the stacked spheres represent the object as an extension of the end effector to represent the surface profile of the extended end effector.

17. The control method for the robotic arm as described in claim 16, characterized in that, The end effector is equipped with a sensor, and the control method further includes: Receive image information of all objects in the workspace of the robotic arm and the robotic arm itself, sent by the image acquisition module; Obtain the feedback information sent by the sensor; The end effector is determined to hold an object based on the image information and the feedback information.

18. The control method for the robotic arm as described in claim 17, characterized in that, The control method further includes: Receive image information of all objects in the workspace of the robotic arm and the robotic arm itself, sent by the image acquisition module; Display the image information on the user interface of the display; and / or, The control commands are generated in response to user actions on the user interface.

19. A control system for a robotic arm, characterized in that, The control system includes one or more processors, the one or more processors being configured to execute a computer program to control the robotic arm to implement the control method as described in any one of claims 1-18.

20. The control system of the robotic arm as described in claim 19, characterized in that, The control system further includes: An image acquisition module is used to acquire image information of all objects in the workspace and the robotic arm, and send the image information to the one or more processors.

21. The control system for the robotic arm as described in claim 20, characterized in that, The control system further includes: A robotic arm, wherein the robotic arm includes a base, at least one link and a joint connecting the at least one link, wherein the at least one link includes an end effector; Sensors mounted on the end effector of the robotic arm; The one or more processors are also used for: Receive image information of all objects in the workspace of the robotic arm and the robotic arm itself, sent by the image acquisition module; Obtain the feedback information sent by the sensor; The end effector is determined to hold an object based on the image information and the feedback information.

22. The control system for the robotic arm as described in claim 21, characterized in that, The control system further includes a display, the display being used for: Display the image information on the user interface; and / or, The user's actions are received through the user interface. The one or more processors are used to generate control instructions according to the operation.

23. An electronic device, characterized in that, include: A memory, wherein the memory stores a computer program; One or more processors, which, when executing the computer program, can control a robotic arm to implement the control method as described in any one of claims 1-18.

24. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by one or more processors, it can control the robotic arm to implement the control method as described in any one of claims 1-18.

25. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by one or more processors, it can control the robotic arm to implement the control method as described in any one of claims 1-18.

Citation Information

Patent Citations

  • Motion control method of mobile mechanical arm in dynamic scene and application thereof

    CN118219260A

  • Differential kinematics control using conformal geometric entity modeling

    US20250121493A1