Method and device for determining control strategy model and method and device for controlling end effector

Abstract

The present disclosure provides a method of determining a control strategy model, a method for controlling an end effector, a device for controlling an end effector, an apparatus for determining a control strategy model, an apparatus for controlling an end effector, an electronic device, a non-volatile computer readable storage medium, and a computer program product. In order to obtain a second control strategy model with generalization and robustness higher than those of the first control strategy model, the first control strategy model is further adjusted through a reinforcement learning scheme to obtain the second control strategy model, and a specific reward score is used in the adjustment process. Under the condition that the control strategy model explores a control scheme as widely as possible, the situation that the control strategy model trained in a simulation environment does not conform to physical constraints and cannot be applied to a physical environment is avoided.

Description

Technical Field

[0001] The present application relates to the field of intelligent robot technology, and specifically to a method for determining a control strategy model, a method for controlling an end effector, a device for controlling an end effector, an apparatus for determining a control strategy model, an apparatus for controlling an end effector, an electronic device, a non-volatile computer-readable storage medium, and a computer program product. Background Art

[0002] Tool manipulation using end effectors is a hot topic in the field of robotics because it involves complex interactions between multiple objects, including the end effector, tool, target object, and the operating environment. For example, a robot needs to use specific tools to grasp and move objects, such as taking objects out of or putting them in a container. The complexity of this process increases significantly with the type of tool, object characteristics, and environmental conditions. The industry is trying to develop end effectors that can adapt to these changes and effectively manipulate tools, but no effective solutions have been proposed so far.

[0003] In order to improve the efficiency and adaptability of robot operations, end-effectors equipped with artificial intelligence models have been proposed to complete the special and difficult task of using end-effectors to operate tools. However, such solutions often require a large amount of training data. The industry is exploring how to improve the generalization ability of neural network models used for end-effectors while reducing training data. In addition, how to improve the control accuracy of the end-effector, increase the number of operable objects, speed up the operation, and enhance anti-interference capabilities are all issues that need to be addressed urgently. By solving these problems, the robot's end-effector can better adapt to changing task scenarios and play a greater role in practical applications. Summary of the invention

[0004] In response to the above problems, the present disclosure provides a method for determining a control strategy model, a method for controlling an end effector, a device for controlling an end effector, an apparatus for determining a control strategy model, an apparatus for controlling an end effector, an electronic device, a non-volatile computer-readable storage medium, and a computer program product.

[0005] According to one aspect of the present disclosure, a method for determining a control strategy model is proposed, wherein the control strategy model is used to control an end effector, wherein the end effector includes a central connecting member and a plurality of multi-joint operating components, and the method includes: obtaining a first control strategy model carried on the end effector, and adjusting the first control strategy model carried on the end effector using a plurality of action rounds to determine a second control strategy model carried on the end effector, wherein, in each of the plurality of action rounds, a learning task corresponding to the action round is determined based on information of the action round; a virtual body corresponding to the end effector simulates and performs the learning task corresponding to the action round using the first control strategy model being adjusted, and determines a reward score corresponding to completing the learning task corresponding to the action round and a reward score related to physical constraints in the process of simulating the execution of the learning task corresponding to the action round; and adjusting the first control strategy model so that the reward score for completing the learning task corresponding to the action round and the reward score related to physical constraints in the process of simulating the execution of the learning task corresponding to the action round are increased.

[0006] According to another aspect of the present disclosure, a method for controlling an end effector is proposed, wherein the end effector includes a central connecting member and a plurality of multi-joint operating components, comprising: at a current moment, acquiring a reference scene image when the end effector performs a target task and a posture corresponding to a contact point between the end effector and a second target tool; utilizing a control strategy model carried on the end effector to generate control information for controlling the end effector at a current time step; determining motor drive information of the central connecting member and the plurality of multi-joint operating components based on the control information for controlling the end effector at the current time step; and controlling the end effector based on the motor drive information of the central connecting member and the plurality of multi-joint operating components; the control strategy model is determined by the above method.

[0007] According to another aspect of the present disclosure, a device for controlling an end effector is proposed, wherein the end effector includes a central connecting member and a plurality of multi-joint operating components, and the device includes: a camera, for acquiring, at a current moment, a reference scene image when the end effector performs a target task; a sensor, for determining, at a current moment, the posture corresponding to the contact point between the end effector and the target tool; a processor, for generating, using a control strategy model carried on the end effector, control information for controlling the end effector at a current time step; and determining, based on the control information for controlling the end effector at the current time step, motor drive information of the central connecting member and the plurality of multi-joint operating components; and a motor, for controlling the end effector based on the motor drive information of the central connecting member and the plurality of multi-joint operating components; wherein the control strategy model is determined by the above method.

[0008] According to another aspect of the present disclosure, a device for determining a control strategy model is proposed, wherein the control strategy model is used to control an end effector, wherein the end effector includes a central connecting piece and multiple multi-joint operating components, and the device includes: a first module, used to obtain a first control strategy model carried on the end effector; a second module, used to adjust the first control strategy model carried on the end effector using multiple action rounds to determine a second control strategy model carried on the end effector, wherein, in each of the multiple action rounds, based on information of the action round, a learning task corresponding to the action round is determined; a virtual body corresponding to the end effector simulates and executes the learning task corresponding to the action round using the first control strategy model being adjusted, and determines a reward score corresponding to completing the learning task corresponding to the action round and a reward score related to physical constraints in the process of simulating the execution of the learning task corresponding to the action round; and adjusting the first control strategy model so that the reward score for completing the learning task corresponding to the action round is increased.

[0009] According to another aspect of the present disclosure, a device for controlling an end effector is proposed, wherein the end effector includes a central connecting member and a plurality of multi-joint operating components, including: a first module, for acquiring, at a current moment, a reference scene image when the end effector performs a target task and a posture corresponding to a contact point between the end effector and a target tool; a second module, for generating control information for controlling the end effector at a current time step by using a control strategy model carried on the end effector; a third module, for determining, based on the control information for controlling the end effector at the current time step, motor drive information of the central connecting member and the plurality of multi-joint operating components; and a fourth module, for controlling the end effector based on the motor drive information of the central connecting member and the plurality of multi-joint operating components; wherein the control strategy model is determined by the above-mentioned method.

[0010] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory, wherein the memory stores a computer executable code, and when the computer executable code is run by the processor, the above method is executed.

[0011] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which executable codes are stored. When the executable codes are executed by a processor, the processor executes the above method.

[0012] According to another aspect of the present disclosure, a computer program product is provided, comprising computer executable instructions, wherein the computer instructions implement the above method when executed by a processor.

[0013] The present disclosure proposes an improved method for determining a control strategy model. Specifically, in order to obtain a second control strategy model with higher generalization and robustness than the first control strategy model, the present disclosure further adjusts the first control strategy model through a reinforcement learning scheme to obtain the second control strategy model, and uses specific reward scores in the adjustment process, namely, "the reward score corresponding to the learning task corresponding to the action round" and "the reward score related to the physical constraints in the process of simulating the execution of the learning task corresponding to the action round". This can ensure that the control strategy model explores the control scheme as widely as possible while avoiding the control strategy model trained in the simulation environment from not meeting the physical constraints, which in turn leads to the inability to be applied in the physical environment.

[0014] In addition, the method for determining the control strategy model proposed in the present disclosure is particularly suitable for special tool operation tasks. Compared with the traditional training scheme for determining the control strategy model suitable for special tool operation tasks, the present disclosure uses a reinforcement learning scheme to adjust the first control strategy model based on the pre-trained first control strategy model. On the one hand, the demand for human demonstration data is reduced, and on the other hand, the training time required for reinforcement learning is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some exemplary embodiments of the present disclosure. For ordinary technicians in this field, other drawings can also be obtained based on these drawings without creative work. The following drawings are not deliberately scaled according to the actual size, and the focus is on illustrating the main purpose of the present disclosure.

[0016] Figure 1 An example task scenario to which the embodiments according to the present disclosure can be applied is schematically shown.

[0017] Figure 2 is a schematic diagram showing an end effector according to an embodiment of the present disclosure.

[0018] Figure 3 is a flowchart illustrating a method for determining a control strategy model according to an embodiment of the present disclosure.

[0019] Figure 4 is a schematic diagram illustrating a method for determining a control strategy model according to an embodiment of the present disclosure.

[0020] Figure 5 is a schematic diagram illustrating a method for controlling an end effector according to an embodiment of the present disclosure.

[0021] Figure 6 is a flowchart illustrating a method for controlling an end effector according to an embodiment of the present disclosure.

[0022] Figure 7 A schematic diagram of a device for controlling an end effector according to some embodiments of the present disclosure is schematically shown.

[0023] Figure 8 An exemplary block diagram of an apparatus for determining a control strategy model according to some embodiments of the present disclosure is schematically shown.

[0024] Fig. 9 An exemplary block diagram of an apparatus for controlling an end effector according to some embodiments of the present disclosure is schematically shown.

[0025] Fig.10 An example block diagram of a computing device according to some embodiments of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present disclosure more obvious, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0027] As shown in the present disclosure and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0028] Although the present disclosure makes various references to certain modules in the apparatus according to the embodiments of the present disclosure, any number of different modules can be used and run on the user terminal and / or server. The modules are only illustrative, and different aspects of the apparatus and method can use different modules.

[0029] Flowcharts are used in this disclosure to illustrate the operations performed by the methods and devices according to the embodiments of the present disclosure. It should be understood that the preceding or following operations are not necessarily performed precisely in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.

[0030] To facilitate description of the present disclosure, concepts related to the present disclosure are introduced below.

[0031] Various embodiments of the present disclosure relate to the design and control of "dexterous hands" in the field of intelligent robots. Dexterous hands are used to simulate the complex movements and functions of human hands, aiming to give robots flexibility and operational capabilities similar to human hands. "Dexterous hands" can perform tasks such as grasping, carrying and operating tools through precise mechanical structures and control systems. The key technologies of "dexterous hands" include multi-degree-of-freedom finger design, integrated sensor systems, and advanced control algorithms, which enable "dexterous hands" to perform precise operations in complex environments.

[0032] The mechanical structure of a "dexterous hand" is usually composed of fingers, palms and connectors with multiple degrees of freedom. Each finger contains multiple joints, which are controlled by servo motors to achieve precise position control and dynamic response. In order to improve the perception ability of the dexterous hand, a variety of sensors are integrated on it, such as IMU sensors and joint angle encoders. These sensors can provide real-time acceleration, posture, joint angle and angular velocity data of the actuator to achieve precise motion control.

[0033] With the development of technology, the application scope of "dexterous hands" is also expanding. In addition to being used for precision assembly and quality inspection in the field of industrial automation, "dexterous hands" also show wide application potential in medical rehabilitation, service robots, education and scientific research. For example, in the field of medical rehabilitation, dexterous hands can assist in surgical operations or be used to develop highly simulated prostheses to help disabled people regain hand function. In the field of service robots, "dexterous hands" enable robots to complete more complex household tasks, such as organizing clothes and preparing food. In addition, with the advancement of artificial intelligence and machine learning technologies, the control strategy of "dexterous hands" is also being continuously optimized to adapt to more complex and dynamic task requirements.

[0034] The solution provided in the embodiments of the present application mainly involves artificial intelligence technology, and applies it to the control field of the robot end effector, which is specifically explained through the following embodiments.

[0035] In summary, the solutions provided by the embodiments of the present disclosure involve technologies such as artificial intelligence and machine learning. The embodiments of the present disclosure will be further described below in conjunction with the accompanying drawings.

[0036] Figure 1 An example scenario 100 is schematically shown in which some embodiments according to the present disclosure may be applied. Figure 1 As shown, the scene 100 includes a robot 110 having an end effector 111, and the end effector 111 can be various types of execution structures, such as a dexterous hand, an end gripper, etc. In addition, the scene 100 also includes a tool 120 and a container 130 containing multiple objects 131. Exemplarily, the control strategy model or tool control model determined according to the scheme provided in some embodiments of the present disclosure can be deployed in the robot 110, such as deployed in a controller network or processor structure for decision-making or control functions in the robot 110, so as to control the end effector 111 of the robot 110 to use the tool 120 to move at least a part of the multiple objects 131 in the container 130 to a target position, such as to a specified height above the container 130.

[0037] exist Figure 1In the figure, the robot 110 is shown as an anthropomorphic robot, the tool 120 is shown as a spoon-shaped tool, the container 130 is shown as a bowl-shaped container, and the object 131 is shown as a spherical object. However, according to the needs of specific application scenarios, the robot 110 may also be other types of robots, such as a robotic arm, etc., the tool 120 may also be other forms of tools, such as other forms of digging tools or other types of tools such as clamping tools, the container 130 may also be a container of other shapes or types, and the object 131 may also be an object of other shapes and other sizes. In addition, optionally, the container 130 may not exist, that is, multiple objects 131 can be directly placed at corresponding positions. And, optionally, the end effector of the robot can also be controlled to move at least a part of the multiple objects 131 to another container.

[0038] Figure 2 is a schematic diagram showing an end effector according to an embodiment of the present disclosure.

[0039] Figure 2 The end effector 111 is further described by taking a “dexterous hand” as an example. Figure 2 The "dexterous hand" (i.e., end effector 111) consists of a central connector and multiple multi-joint operating components, each finger consisting of multiple segments, each segment equipped with at least one active bending joint. These joints are typically controlled by servo motors that provide precise position control and dynamic response, enabling the dexterous hand to perform delicate operations. The design of the joints allows for a wide range of movements, including extension and flexion, thereby enabling diverse grasping modes and operating capabilities. In addition, the mechanical structure of the dexterous hand also includes lightweight but strong materials, such as aluminum alloy or carbon fiber, to ensure strength and durability, while reducing weight and improving operational flexibility and responsiveness.

[0040] In order to enhance the perception and feedback capabilities of the "dexterous hand", a variety of sensors are integrated on its central connection. These include an inertial measurement unit (IMU) sensor, which measures the linear acceleration and angular velocity of the actuator by integrating accelerometers and gyroscopes, providing data on the posture and motion of the end effector. These data are helpful for performing precise motion control, especially when complex operations are required or working in a dynamic environment. In addition, each joint is optionally equipped with a joint angle encoder to provide feedback on the joint position and velocity, thereby achieving precise motion control. The integration of these sensors enables the dexterous hand to perform adaptive control to cope with unpredictable external disturbances and changes.

[0041] The control system of the "dexterous hand" is responsible for coordinating the actions of the motors to achieve complex manipulation tasks. Through advanced algorithms, the "dexterous hand" is able to simulate the actions of the human hand, such as grasping, carrying objects, and using tools. This ability makes the dexterous hand promising for application in automated production lines, robotic surgical assistance, service robots, and scientific research. The control system usually includes a real-time operating system and advanced control algorithms, such as PID control, model predictive control, or machine learning algorithms, to achieve highly precise and adaptable operations.

[0042] Further, Figure 2 The dexterous hand in the paper can be used to simulate the human hand to use tools / operate tools. In daily life, the use of tools is essential to extending human physical capabilities. For example, tools such as spoons and screwdrivers enable people to perform tasks such as scooping granules or liquids and tightening screws, which are difficult to complete without the use of tools. The design of the "dexterous hand" enables it to imitate these basic human actions, and through integrated sensors and sophisticated control systems, it can operate tools with a high degree of dexterity and precision.

[0043] However, it is often difficult for the end effector to manipulate the tool to control the target object. Specifically, the interaction between the end effector, the tool, and the target object or environment is very complex. This complex interaction relationship means that the method designed for the end effector to directly manipulate the object cannot be simply applied to the task of manipulating the object using the tool (i.e., the tool manipulation task). The tool manipulation task requires the end effector to understand and adapt to the dynamic characteristics of the tool and the interaction with the target object, which requires the improvement and adjustment of the overall control strategy of the end effector.

[0044] In addition, the end effector is usually not in direct contact with the target object or environment during tool manipulation, which leads to insufficient observation and feedback of the contact status between the tool and the target object. The lack of such direct feedback makes it more difficult to achieve effective feedback control. Tool manipulation tasks usually involve multiple target objects and complex physical constraints, which makes control methods based on traditional dynamic models difficult to deploy. Therefore, in order to achieve effective tool manipulation in the end effector, it is necessary to develop improved control methods to adapt to the complex relationship between the end effector, the tool and the target object or environment, and to address the challenges caused by the lack of direct contact and the involvement of multiple objects and physical constraints.

[0045] Currently, in order to overcome the challenges in tool operation, data-driven methods have become a hot topic of research. Imitation learning methods have shown potential in this regard. Imitation learning schemes guide robot learning by analyzing the behavior of human operators (i.e., human demonstrations). However, imitation learning schemes usually rely on humans to directly operate or remotely control the robot's end effector during the data collection process, which is not only time-consuming but also inefficient.

[0046] To reduce the reliance on human demonstrations, reinforcement learning provides an alternative that allows robots to autonomously learn skills through interaction with the environment with little or no direct human guidance. Although reinforcement learning methods have achieved certain results, the diversity and complexity of tool manipulation tasks make the training process of reinforcement learning challenging. For example, in scooping tasks, the robot's motion strategy needs to be adjusted according to factors such as the shape of the spoon and bowl, the amount and material of the target contents, etc. In order to train a general strategy that can adapt to various scenarios, a large amount of training data is often required to ensure that the neural network model carried by the end effector can effectively learn from the training data.

[0047] Therefore, in response to the above problems, according to one aspect of the present disclosure, a method for determining a control strategy model is proposed, wherein the control strategy model is used to control an end effector, and the end effector includes a central connecting member and a plurality of multi-joint operating components. The method includes: obtaining a first control strategy model carried on the end effector, and adjusting the first control strategy model carried on the end effector using a plurality of action rounds to determine a second control strategy model carried on the end effector, wherein, in each of the plurality of action rounds, based on information of the action round, a learning task corresponding to the action round is determined; a virtual body corresponding to the end effector simulates and executes the learning task corresponding to the action round using the first control strategy model being adjusted, and determines a reward score corresponding to completing the learning task corresponding to the action round and a reward score related to physical constraints in the process of simulating the execution of the learning task corresponding to the action round; and adjusting the first control strategy model so that the reward score for completing the learning task corresponding to the action round and the reward score related to physical constraints in the process of simulating the execution of the learning task corresponding to the action round are increased.

[0048] According to one aspect of the present disclosure, a method for controlling an end effector is also proposed, wherein the end effector includes a central connecting member and a plurality of multi-joint operating components, comprising: at a current moment, obtaining a reference scene image when the end effector performs a target task and a posture corresponding to a contact point between the end effector and a second target tool; utilizing a control strategy model carried on the end effector to generate control information for controlling the end effector at a current time step; determining motor drive information of the central connecting member and the plurality of multi-joint operating components based on the control information for controlling the end effector at the current time step; and controlling the end effector based on the motor drive information of the central connecting member and the plurality of multi-joint operating components; wherein the control strategy model is determined by the above method.

[0049] In order to obtain a second control strategy model with higher generalization and robustness than the first control strategy model, the present invention further adjusts the first control strategy model through a reinforcement learning scheme to obtain the second control strategy model, and uses specific reward scores in the adjustment process, namely, "the reward score corresponding to the learning task corresponding to the action round" and "the reward score related to the physical constraints in the process of simulating the execution of the learning task corresponding to the action round". This can ensure that the control strategy model explores the control scheme as widely as possible while avoiding the control strategy model trained in the simulation environment from not meeting the physical constraints, which in turn makes it impossible to be applied in the physical environment.

[0050] The method for determining the control strategy model proposed in the present disclosure is particularly suitable for special tool operation tasks. Compared with the traditional training scheme for determining the control strategy model suitable for special tool operation tasks, the present disclosure uses a reinforcement learning scheme to adjust the first control strategy model based on the pre-trained first control strategy model. On the one hand, the demand for human demonstration data is reduced, and on the other hand, the training time required for reinforcement learning is reduced.

[0051] Specifically, in the method provided by the present disclosure, only training data for training the first control strategy model needs to be collected, which significantly reduces the amount of data required for training compared to traditional solutions. At the same time, in the method provided by the present disclosure, the second control strategy model is obtained by adjusting the first control strategy model to adapt to the second target task different from the first target task, which significantly improves the generalization of the control strategy model compared to traditional solutions. At the same time, since the reinforcement learning is performed on the basis of the first control strategy model, the control strategy model does not need to conduct too much ineffective exploration in this process, which greatly reduces the time required for the reinforcement learning process.

[0052] In some embodiments of the present disclosure, imitation learning and reinforcement learning can also be combined so that the neural network model carried by the end effector can perform imitation learning from fewer human demonstrations, and acquire the ability to complete tool operation tasks through reinforcement learning on the basis of imitation learning. As a result, the present disclosure can significantly reduce the need to collect human demonstration data, so that the neural network model carried by the end effector can quickly perform imitation learning through a very small amount of training data and can work effectively in unseen scenes using reinforcement learning. Through the embodiments of the present disclosure, the neural network model carried by the end effector can extract key information from limited training data and generalize it to new and unknown environments. The embodiments of the present disclosure not only improve learning efficiency, but also enhance the adaptability and flexibility of robots in diverse tasks.

[0053] Next, refer to Figure 3 The various embodiments of the present disclosure are described in an integrated manner.

[0054] Figure 3 is a flow chart illustrating a method 30 of determining a control strategy model according to an embodiment of the present disclosure.

[0055] Method 30 can be used to determine a control strategy model (e.g., the first control strategy model and the second control strategy model described in detail below). The control strategy model is used to control the end effector to perform a target task (e.g., the first target task and the second target task described in detail below) using a target tool, and the target task may include using a target tool (e.g., the first target tool and the second target tool described in detail below) to move at least one target object (e.g., the first target object and the second target object described in detail below) from an initial position (e.g., the first initial position and the second initial position described in detail below) to a target position (e.g., the first target position and the second target position described in detail below). Of course, the present disclosure is not limited to this.

[0056] For example, the method 30 may be used to control Figure 1 and Figure 2 The end effector 111 of the robot 110 shown is used to control the end effector 111 of the robot 110 to use the tool 120 to move the object 131 in the container 130. Optionally, the method 30 can be implemented in a real machine environment or in a virtual environment (or simulation environment). Optionally, the target tool can be any type of tool, such as a digging tool, a clamping tool, a sticking tool, a suction tool, a fork tool, etc., and the target object can be an object of any shape, such as a spherical object, a block object, an irregularly shaped object, etc. Of course, the present disclosure is not limited to this.

[0057] The initial position should be understood as the position where multiple target objects are located before starting to perform the target task. For example, multiple target objects can be placed in a container at the initial position, or can be directly scattered or piled at the initial position. Similarly, the target position should be understood as the position to which the target object is expected to be moved. For example, it can be expected that at least one of the multiple target objects can be moved to a container at the target position, or it can be expected to be moved or placed at the target position. It should be understood that the initial position or target position can refer to a position range, not necessarily a precise position point. Of course, the present disclosure is not limited to this.

[0058] The method 30 for determining a control strategy model according to an embodiment of the present disclosure may include: Figure 3 Steps S301 to S302 are shown. Of course, method 30 may also include more or fewer operations, and the present disclosure is not limited thereto. Optionally, step S301 may be an operation in the pre-training phase. Operation S302 is an operation in the reinforcement learning phase.

[0059] As described above, the end effector includes a central connecting member and a plurality of multi-joint operating components (such as mechanical fingers). The central connecting member and the plurality of multi-joint operating components can refer to Figure 1 and Figure 2 The relevant description will not be repeated in this disclosure.

[0060] In step S301, a first control strategy model carried by the end effector is obtained.

[0061] Optionally, the first control strategy model is used to control the end effector to perform a first target task, wherein, during the process of the end effector performing the first target task, the end effector uses a first target tool to move a first target object from the first initial position to the first target position. Optionally, during the process of the first target object moving from the first initial position to the first target position, the first target tool provides a force to the first target object, and the end effector does not contact the first target object. Of course, the present disclosure is not limited to this.

[0062] Optionally, the first control strategy model can be a neural network model with a preset network structure, and its network structure can be constructed based on an existing neural network structure in the relevant technology, or it can also be any new network structure. In addition, optionally, the first control strategy model can also be other types of models. The present disclosure does not specifically limit the specific model type and structure of the first control strategy model.

[0063] Optionally, the first control strategy model is a pre-trained neural network model. The first control strategy model can be pre-trained in a variety of ways, including but not limited to transfer learning, imitation learning, and self-supervised learning. The following uses imitation learning as an example to briefly introduce the process of "obtaining the first control strategy model carried by the end effector", and those skilled in the art should understand that the present disclosure is not limited to this.

[0064] Optionally, step S301 includes: obtaining a reference scene image when the end effector performs a first target task and a reference trajectory corresponding to a contact point between the end effector and a first target tool; based on the reference scene image when the end effector performs the first target task and the reference trajectory corresponding to a contact point between the end effector and the first target tool, determining a first control strategy model carried by the end effector, wherein the first control strategy model is used to control the end effector to perform the first target task. The present disclosure is not limited thereto.

[0065] Optionally, the reference scene image when the end effector performs the first target task is not limited to one image, but may include multiple images or even a video. These reference scene images may show the scene at different moments in the process of the end effector performing the first target task to provide more comprehensive information, thereby helping the end effector to better understand and identify the progress of performing the target task. Figure 1 In the scene of using a spoon to scoop up particles, the reference scene images at different times show the different relative positions and states of the bowl and spoon. These images continuously show the progress of the task.

[0066] Optionally, the reference scene image when the end effector performs the first target task is a depth image captured by a depth camera. The depth camera calculates the distance of the object and generates a depth image by emitting infrared light and measuring the time of the reflected light. Optionally, the reference scene image includes three-dimensional information of the scene in which the end effector performs the first target task. Thus, the first strategy model carried on the end effector can perform accurate spatial positioning and object recognition based on the reference scene image, thereby enabling the robot to better understand its environment and determine the target task to be performed. Of course, the present disclosure is not limited to this.

[0067] Optionally, the reference trajectory corresponding to the contact point between the end effector and the first target tool is the motion trajectory of the contact point when the end effector performs the first target task. The reference trajectory may include at least one of the posture information, position information, velocity information, acceleration information, angular velocity information, and angular acceleration information of the contact point at each time step. Optionally, the reference trajectory can be collected by a sensor on the end effector. Among them, the position information can be a three-dimensional coordinate in a specified coordinate system, such as the three-dimensional coordinate of the contact point, and the posture information can be described by a quaternion. Alternatively, the posture of the target tool can be described in other ways, or the number of parameters can be adjusted according to the actual allowed degrees of freedom. Optionally, the position change or the posture change can be described in the same way as the posture of the target tool. Of course, the present disclosure is not limited to this.

[0068] In one example, the reference trajectory can be represented by a time series numerical sequence composed of the position information of the contact point corresponding to each time step. Optionally, each element in the time series numerical sequence can have multiple dimensions, which respectively represent the position of the contact point in the x-axis direction, the position of the contact point in the y-axis direction, the position of the contact point in the z-direction (gravity direction) corresponding to a certain time step, and the posture information represented by quaternion. Of course, the reference trajectory can also be represented by other data structures, and the present disclosure is not limited to this.

[0069] In one example, the reference scene image and the reference trajectory can be acquired in the following manner. In the real physical space, the end effector is controlled by a human operator to operate the first target tool to perform the first target task. In this process, the human operator controls the end effector directly or indirectly to interact with the selected first target tool (such as a spoon, a screwdriver, pliers, etc.) to complete a specific target task (such as scooping particles or liquids, tightening screws, clamping objects, etc.). The reference scene image and the reference trajectory corresponding to the contact point between the end effector and the first target tool can be captured in real time by a depth camera deployed in the real physical space and a sensor deployed on the end effector. Of course, the present disclosure is not limited to this.

[0070] In another example, the reference scene image and the reference trajectory can also be obtained in the following manner. In a simulation environment, a human operator draws an animation to control a virtual end effector to operate a virtual first target tool to perform a first target task. In the animation, the end effector is controlled to interact with a virtual first target tool (such as a virtual spoon, a virtual screwdriver, a virtual pliers, etc.) to complete a specific target task (such as scooping virtual particles or liquids, tightening virtual screws, clamping virtual objects, etc.). The reference scene image and the reference trajectory corresponding to the contact point between the end effector and the first target tool can be captured in real time by deploying a perspective camera in the simulation environment. Of course, the present disclosure is not limited to this.

[0071] Optionally, the process of determining the first control strategy model carried by the end effector is the process of training the first control strategy model. In this training process, the reference scene image when the end effector performs the first target task and the reference trajectory corresponding to the contact point between the end effector and the first target tool are used as training data, and the model parameters of the first control strategy model are continuously adjusted until a predetermined number of iterations is reached or the model parameters converge. Thus, the first control strategy model attempts to enable the end effector to move autonomously through its own motor without the control of a human operator, so that the end effector can control the first target tool to move approximately along the reference trajectory to move the first target object from the first initial position to the first target position or near the first target position.

[0072] Optionally, as described above, the reference scene image and the reference trajectory can be acquired in both the physical space and the simulation environment. The reference scene image and the reference trajectory provide reference information for the first control strategy model, so that it can gradually adjust the control information used to control the end effector. The control information used to control the end effector indicates the action that the end effector should perform at the current time step. By performing the action, the end effector can make the desired posture of the contact point between the end effector and the first target tool roughly match the reference trajectory at the next time step.

[0073] Although the training process of the first control strategy model can adopt the method of imitation learning to ensure that the first control strategy model learns a rough control strategy that can achieve the first target task in the limited training data. However, due to the inherent limitations of imitation learning, once the real scene changes or any parameter in the target task changes (for example, the target tool, the target object, the initial position of the target object, or the target position of at least one of the changes), the performance of the first control strategy model trained by imitation learning will be poor. Therefore, the first control strategy model will be adjusted (or fine-tuned) through step S302 to obtain a control strategy model (i.e., the second control strategy model) with higher generalization and robustness.

[0074] In step S302, the first control strategy model carried on the end effector is adjusted using multiple action rounds to determine the second control strategy model carried on the end effector. In each of the multiple action rounds, the learning task corresponding to the action round is determined based on the information of the action round; the virtual body corresponding to the end effector simulates the execution of the learning task corresponding to the action round using the first control strategy model being adjusted, and determines the reward score corresponding to the completion of the learning task corresponding to the action round and the reward score related to the physical constraints in the process of simulating the execution of the learning task corresponding to the action round; and the first control strategy model is adjusted so that the reward score for completing the learning task corresponding to the action round and the reward score related to the physical constraints in the process of simulating the execution of the learning task corresponding to the action round are increased.

[0075] Optionally, the second control strategy model carried on the end effector is used to control the end effector to perform a second target task, wherein, during the process of the end effector performing the second target task, the end effector uses the second target tool to move the second target object from the second initial position to the second target position, and the first target task and the second target task are different.

[0076] Optionally, during the process of the second target object moving from the second initial position to the second target position, the second target tool provides a force to the second target object, and the end effector does not contact the second target object. Of course, the present disclosure is not limited thereto.

[0077] Optionally, the first target task and the second target task meet at least one of the following conditions: the first target tool is different from the second target tool; the first target object is different from the second target object; the number of the first target objects is different from the number of the second target objects; the first initial position is different from the second initial position; the first target position is different from the second target position; or the first target object is located in a first container, the second target object is located in a second container, and the first container is different from the second container. Of course, the present disclosure is not limited to this.

[0078] Optionally, in order to obtain a second control strategy model with higher generalization and robustness than the first control strategy model, the second control strategy model can be the first control strategy model fine-tuned using reinforcement learning, and the second control strategy model has the same architecture as the first control strategy model, but the second control strategy model has higher generalization than the first control strategy model. The training process of reinforcement learning allows the second control strategy model to learn more environmental dynamics and potential strategies, so that it can make more reasonable decisions when facing new situations.

[0079] Specifically, in the reinforcement learning process, by learning the same or different learning tasks in each action round, the obtained second control strategy model can explore as many control strategies as possible, making the control strategy model more "intelligent". Optionally, in the process of the virtual body corresponding to the end effector simulating the execution of the learning task corresponding to the action round, the end effector simulates the execution of: using the target tool specific to the action round to move the target object specific to the action round from the initial position specific to the action round to the target position corresponding to the action round.

[0080] Optionally, in the process of reinforcement learning, the idea of ​​human learning courses can be used for reference. Specifically, in the process of learning a skill in school, humans often need to consolidate the knowledge they have learned and improve their performance through reasonably set courses, and then learn higher-level skills. Generally speaking, humans need to gradually adapt to more complex environments by learning courses from easy to difficult, where courses can be a development concept, which can be composed of certain educational goals, specific knowledge and experience, and expected learning activities. It contains a rich, basic, creative and potential set of plans and settings. Based on this, the present disclosure proposes that a set of designed processes can be used to adapt the control strategy model from easy to difficult to different task difficulty scenarios, and finally converge to a model that can complete difficult tasks. Inspired by the human course learning process, the present disclosure introduces the idea of ​​course learning into the training process of reinforcement learning to solve the problem that the control strategy model lacks exploratoryness after learning the basic strategy, or is difficult to converge when directly facing difficult tasks.

[0081] Optionally, in S302, based on the information of the action round including the course difficulty information and course progress information corresponding to the action round, a target tool specific to the action round or a target object specific to the action round is determined.

[0082] Optionally, in order to enhance the robustness of reinforcement learning, the specific initial position of the action round and the target position corresponding to the action round may also be randomly determined. Of course, the present disclosure is not limited to this.

[0083] The course difficulty information corresponding to the current action round is associated with the course progress information of the current action round, and the course progress information indicates that the later the action round is executed, the higher the course difficulty information corresponding to the current action round is. Of course, the present disclosure is not limited to this.

[0084] Optionally, in some embodiments of the present disclosure, the course difficulty information may also be dynamically adjusted. For example, the course difficulty information corresponding to the current action round may be associated with the reward score of the learning task corresponding to the previous action round. The higher the reward score of the learning task corresponding to the previous action round, the higher the course difficulty information corresponding to the current action round. Of course, the present disclosure is not limited to this.

[0085] Optionally, in some embodiments of the present disclosure, the same course difficulty information corresponds to multiple action rounds, and different action rounds correspond to different learning tasks, wherein the different learning tasks corresponding to different action rounds satisfy at least one of the following conditions: different action rounds have different specific target tools; different action rounds have different specific target objects; different action rounds have different numbers of specific target objects; or different action rounds have specific target objects located in different containers. In addition, in the learning tasks corresponding to different action rounds, the specific initial positions of different action rounds may also be different, or the specific target positions of different action rounds may also be different. Of course, the present disclosure is not limited to this.

[0086] Thus, in the process of reinforcement learning, the model parameters of the preset model can be updated based on the execution of the learning task, so that it converges towards the direction of successfully completing the learning task. When the course progress information indicates that the number of successful executions of the learning task within multiple consecutive action rounds meets the threshold condition, the course difficulty information can be updated, and then the next stage of model training operations can be performed based on the updated course difficulty information. Thus, the number of targets can be gradually increased to gradually increase the course difficulty, thereby gradually guiding the virtual body corresponding to the end effector to complete higher difficulty target tasks through exploration, avoiding the problem that the control strategy model converges to only completing low difficulty tasks, or the exploration space is too large to converge.

[0087] In addition, since the reinforcement learning process is implemented in a simulation environment, this solution can avoid real machine loss as much as possible. Optionally, "the reward score corresponding to the learning task corresponding to the action round" refers to the reward score obtained by the virtual body corresponding to the end effector after completing a specific learning task in the virtual environment, and "the reward score related to the physical constraints in the process of simulating the execution of the learning task corresponding to the action round" refers to the reward score obtained by the virtual body corresponding to the end effector when performing the task in the simulation environment, taking into account the physical constraints. Both of these reward scores are sparse rewards, which can prompt the control strategy model to conduct more extensive exploration in the simulation environment to find behaviors that may lead to rewards, thereby possibly discovering new and more effective strategies.

[0088] By "completion of the reward score corresponding to the learning task corresponding to the action round", the virtual body corresponding to the end effector must learn from limited feedback, which forces the algorithm to use samples more efficiently, thereby improving the sample efficiency of the learning process. At the same time, because the virtual body corresponding to the end effector is exposed to more diverse situations during training, it is more likely to develop strategies that can generalize to new situations, rather than being limited to specific behaviors that often bring rewards.

[0089] For example, the completion of the scooping action can be evaluated by the following conditions: first, whether the target tool specific to the action round (i.e., the virtual body corresponding to the spoon) is directly above the virtual body corresponding to the container; second, whether the amount of objects scooped in the virtual body corresponding to the spoon (i.e., the number of target objects specific to the action round) meets the predetermined standard. Specifically, the sufficiency of the amount of objects needs to be determined by comprehensively considering factors such as the capacity of the spoon, the volume of the object, and the height of the contents in the spoon. Through these specific conditions, it can be effectively determined whether the virtual body corresponding to the end effector has completed the corresponding learning task.

[0090] Optionally, the "reward score corresponding to completing the learning task corresponding to the action round" can be set to indicate at least one or a combination of the following items: whether the end effector successfully moves the target object specific to the action round from the initial position specific to the action round to the target position corresponding to the action round using the target tool specific to the action round; or whether the target object overflows during the process in which the virtual body corresponding to the end effector simulates the execution of the learning task corresponding to the action round. Of course, the present disclosure is not limited to this.

[0091] By “simulating the reward scores related to physical constraints during the learning task corresponding to the action round”, the virtual body corresponding to the end effector can only perform actions that meet the physical constraints of the environment and prevent undesirable training results. For example, for the scooping action, due to the difference between the simulation environment and the physical space, the actions simulated by the virtual body corresponding to the end effector in the simulation environment may not meet the physical constraints. These actions are impossible to achieve in the real space. Therefore, it is necessary to judge whether the virtual body corresponding to the spoon is correctly inserted into the virtual body corresponding to the container, whether the virtual body corresponding to the spoon and the virtual body corresponding to the target object are interlaced, and so on. These need to be circumvented by designing corresponding reward scores to ensure the effectiveness and safety of training.

[0092] Optionally, the "reward score related to the physical constraints during the simulation of the learning task corresponding to the action round" can be set to indicate at least one or a combination of the following items: whether the target tool specific to the action round collides with the target object specific to the action round during the simulation of the learning task corresponding to the action round by the virtual body corresponding to the end effector; whether the target tool specific to the action round collides with the container where the target object specific to the action round is located during the simulation of the learning task corresponding to the action round by the virtual body corresponding to the end effector; whether the end effector is separated from the target tool during the simulation of the learning task corresponding to the action round by the virtual body corresponding to the end effector; and whether the target tool exceeds the observation range during the simulation of the learning task corresponding to the action round by the virtual body corresponding to the end effector. Of course, the present disclosure is not limited to this.

[0093] Specifically, the training process of the reinforcement learning can be briefly described as follows: based on the scene image at the current time step during the process of the end effector performing the second target task and the posture corresponding to the contact point between the end effector and the second target tool, the control information of the end effector at the current time step is determined using the first control strategy model: at least partially based on the control information of the end effector at the current time step, a reward score for the control information of the end effector at the current time step is determined; and at least partially based on the reward score for the control information of the end effector at the current time step, the model parameters of the first control strategy model are updated to determine the second control strategy model carried by the end effector. Reference will be made later Figure 4 The process is further described, and the disclosure is not repeated here. Of course, the disclosure is not limited to this.

[0094] Through the training process of reinforcement learning, the second control strategy model determines the control information corresponding to each time step, and then learns according to the feedback (reward or punishment). As the reinforcement learning process deepens, the second control strategy model learns how to switch between different states to achieve long-term goals, improving its generalization ability. Through reinforcement learning, the second control strategy model not only inherits the knowledge learned by the first control strategy model, but also enhances flexibility and adaptability.

[0095] Therefore, the present disclosure proposes an improved method for determining a control strategy model. Specifically, in order to obtain a second control strategy model with higher generalization and robustness than the first control strategy model, the present disclosure further adjusts the first control strategy model through a reinforcement learning scheme to obtain the second control strategy model, and uses specific sparse reward scores in the adjustment process, namely, "the reward score corresponding to the learning task corresponding to the action round" and "the reward score related to the physical constraints in the process of simulating the execution of the learning task corresponding to the action round". This can ensure that the control strategy model explores the control scheme as widely as possible while avoiding the control strategy model trained in the simulation environment from not meeting the physical constraints, which in turn leads to the inability to be applied in the physical environment.

[0096] In addition, the method for determining the control strategy model proposed in the present disclosure is particularly suitable for special tool operation tasks. Compared with the traditional training scheme for determining the control strategy model suitable for special tool operation tasks, the present disclosure uses a reinforcement learning scheme to adjust the first control strategy model based on the pre-trained first control strategy model. On the one hand, the demand for human demonstration data is reduced, and on the other hand, the training time required for reinforcement learning is reduced.

[0097] Furthermore, the present disclosure combines the ideas of reinforcement learning with curriculum learning, so that the control strategy model carried by the end effector can adapt to the changing operating environment and task requirements during the reinforcement learning process, and achieve more flexible and efficient operating performance.

[0098] It is worth noting that in some embodiments of the present disclosure, reinforcement learning is also combined with imitation learning, thereby significantly reducing the need to collect human demonstration data, so that the control strategy model carried by the end effector (such as any one of the first control strategy model and the second control strategy model mentioned above) can quickly perform imitation learning through a small amount of training data and can work effectively in unseen scenarios using reinforcement learning. Through the embodiments of the present disclosure, the control strategy model carried by the end effector can extract key information from limited training data and generalize it to new and unknown environments. The embodiments of the present disclosure not only improve learning efficiency, but also enhance the adaptability and flexibility of the robot in diverse tasks.

[0099] Next, refer to Figure 4 The details of the embodiments of the present disclosure are further described.

[0100] Figure 4 is a schematic diagram illustrating a method 30 for determining a control strategy model according to an embodiment of the present disclosure.

[0101] Optionally, the embodiment of the present disclosure makes architectural improvements to the control strategy model. Compared with the structure of the traditional strategy model, the improved control strategy model (i.e., the first control strategy model and the second control strategy model) is composed of two decoupled modules: a visual encoder and a controller network. In the process of adjusting the first control strategy model to obtain the second control strategy model, a value network decoupled from both the "visual encoder and the controller network" can also be introduced to assist in the adjustment of the first control strategy model. In addition, the visual encoder may also be combined with an encoder for encoding sensory information, such as an encoder for tactile information, to better capture scene information and progress information of the target task execution. Of course, the present disclosure is not limited to this.

[0102] Optionally, the visual encoder processes the input reference scene image and extracts a visual encoding feature vector. The visual encoding feature vector incorporates the relative pose information between the end effector and the target tool and the progress information of the target task. The visual encoder converts the reference scene image into a machine-understandable form to provide the controller network with the information required for decision-making. The controller network calculates the pose change corresponding to the target tool at each time step based on the visual encoding features and the reference trajectory corresponding to the contact point between the end effector and the target tool.

[0103] refer to Figure 4, in the imitation learning stage, in step S301, the first control strategy model can be obtained by training the first control strategy model. Optionally, as a specific example, in the imitation learning stage, the reference scene image when the end effector performs the first target task and the reference trajectory corresponding to the contact point between the end effector and the first target tool can be first collected in the physical space for training. Then, in the simulation environment, the data collected in the physical space is projected, and the data projected into the simulation environment is amplified, so as to collect relatively more training data while minimizing human demonstration. Of course, the training data may not be amplified in the simulation environment, and the present disclosure is not limited to this.

[0104] Optionally, it is assumed that in the physical space, a reference scene image when the end effector performs the first target task and a reference trajectory corresponding to the contact point between the end effector and the first target tool are obtained. Then, the reference scene image when the end effector performs the first target task in the physical space and the reference trajectory corresponding to the contact point between the end effector and the first target tool can be projected into the simulation environment to generate projection data of the reference scene image in the simulation environment and projection data of the reference trajectory projected in the simulation environment. The first control strategy model carried on the end effector is trained so that the first control strategy model can generate control information for controlling the end effector at the current time step based on the projection data of the reference scene image in the simulation environment and the projection data of the reference trajectory projected in the simulation environment. The control information for controlling the end effector indicates the action performed by the end effector at the current time step.

[0105] Specifically, reference scene images and reference trajectories captured in the physical space can be obtained through sensors and cameras. These data are preprocessed, such as scaling, cropping, and normalization, to adapt to the input requirements of the simulation environment. Next, a simulation environment that simulates the original physical scene as much as possible is built in MuJoCo. MuJoCo is an advanced physics engine for simulating multi-joint characters and rigid body dynamics, including contact and collision. On this basis, trajectory models in the simulation environment are designed according to the actual captured reference trajectories, which can be used as targets for agent training or as dynamic elements in the environment. Subsequently, the processed reference scene images and reference trajectories can be projected into the simulation environment, which involves the combination of image rendering and physical simulation. In this process, OpenAI Gym, as a toolkit for developing and comparing reinforcement learning algorithms, provides a unified interface to describe the interaction between the environment and the agent. Gym supports a variety of predefined environments, including simulation environments that use MuJoCo as the backend, which facilitates imitation training.

[0106] More specifically, a MuJoCo XML file can be written to perform physical simulation modeling. For example, the corresponding model can be loaded into the MuJoCo environment through an XML file. For example, for a scene in which a scooping tool such as a spoon scoops a target object (such as a small ball) from a target container (such as a bowl), the virtual bodies corresponding to the end effector, spoon, and bowl (referred to as a virtual end effector, a virtual spoon, and a virtual bowl, respectively) can be loaded, and the virtual end effector and the virtual spoon can be fixed together, wherein the virtual end effector can be regarded as an actuator for controlling the virtual spoon. The welding point between the virtual end effector and the virtual spoon can be regarded as corresponding to the contact point between the end effector and the spoon in the real machine environment. Since the spoon and the bowl are typical non-convex bodies with complex geometric shapes, they cannot be directly implemented in the virtual environment. An embodiment of the present disclosure can slice the scanned real spoon and bowl models into multiple convex hulls, and then recombine them into a complete object.

[0107] In addition, several small balls can be added to the environment through XML files. The MujocoEnv provided by OpenAI Gym can be used to access the written MuJoCo environment. MujocoEnv can be created to define environment description variables such as action_space and observation_space, and implement functional functions such as init (initialization), step, reset, etc. MujocoEnv can correspond to the written MuJoCo environment. action_space can be used to define action information, and the action information may include the change in the pose of the target tool, for example, it can be represented by 7 bits of information. observation_space can be used to define model input information, and the input information may include a reference trajectory composed of the pose information of the contact point, the relative position relationship between multiple target objects and the target tool, etc. In order to determine the relative posture between the target tool and the target object, the embodiment of the present disclosure may also set multiple position points on the target tool to determine whether the target object is accurately located on the target tool.

[0108] For example, for the scenario of a spoon scooping a ball, 11 input parameters can be defined, of which 7 represent the parameters of the spoon's posture, 3 unique codes represent the positional relationship between the ball and the spoon, and 1 represents the height of the topmost ball in the initial state (i.e., the liquid level). In the unique code design, 100 is used to indicate that there is no ball in the spoon, 010 represents that there is a ball in the spoon, and 001 represents that there is a ball in the spoon and the center of mass of the ball is higher than the center of mass of the spoon. Alternatively, 18 input parameters can be defined. In addition to the above parameters, there can also be 7 parameters representing the posture of the contact point between the virtual spoon and the virtual end effector. For another example, for the scenario of a clamp grabbing a block, 11 input parameters can be defined, of which 7 represent the parameters of the clamp's posture, 1 represents the parameter of the clamp's opening and closing angle, 2 unique codes represent the positional relationship between the target object and the clamp, and 1 represents the task scenario. There are two types of positional relationships between the target object and the clamp, namely, being clamped and not being clamped.

[0109] As mentioned above, the first control strategy model carried by the end effector includes a visual encoder and a controller network. The visual encoder and the controller network can be trained separately. The input of the visual encoder is the reference scene image corresponding to a single moment, and the output is the corresponding visual encoding vector, which can capture the key features of the reference scene image and provide the controller network with information such as the execution progress and current status of the current target task.

[0110] To facilitate training in a simulation environment and subsequent deployment on a real machine, the visual encoder should be able to encode both the reference scene image in the physical space and the reference scene image in the simulation environment, and output the same visual encoding vector for the same scene. The visual encoder will serve as a bridge for information conversion between the simulation environment and the physical space. To this end, optionally, the following scheme can be used to train the visual encoder: using the visual encoder in training, encode the reference scene image when the end effector performs the first target task in the physical space to generate a visual encoding vector; using the visual encoder in training, encode the projection data of the reference scene image in the simulation environment to generate visual label data; adjust the model parameters of the visual encoder so that the difference between the visual encoding vector and the visual label data converges. In this way, the visual encoder can output the same visual encoding vector for the same scene in both the physical space and the simulation environment, thereby providing accurate input data to the controller network. Of course, the present disclosure is not limited to this.

[0111] Next, in step S301, after completing the training of the visual encoder, the controller network can be trained separately. Optionally, the training process includes: based on the projection data of the reference scene image corresponding to the current time step in the simulation environment, using the trained visual encoder, generating a visual encoding vector corresponding to the current time step; based on the visual encoding vector corresponding to the current time step and the true value of the pose data of the contact point at the current time step in the projection data of the reference trajectory projected in the simulation environment, using the controller network in training to predict the predicted value of the pose data of the contact point at the next time step; and adjusting the model parameters of the controller network in training so that the difference between the predicted value of the pose data of the contact point at the next time step and the true value of the pose data of the contact point at the next time step in the projection data of the reference trajectory projected in the simulation environment converges. Of course, the present disclosure is not limited to this.

[0112] It is worth noting that the pose data of the contact point at the next time step can be used to indicate the control information for controlling the end effector at the current time step. Specifically, the difference between the pose of the contact point at the next time step and the pose at the current time step will determine how the motors of the end effector should be driven at the current time step. In other words, if the pose data of the contact point between the end effector and the target tool at the next time step can be accurately predicted, it is possible to know how to control the end effector.

[0113] As an example, assume that the visual encoding vector corresponding to the current time step can be represented as an 8-dimensional feature vector O visual Subsequently, this visual encoding vector is combined with the 7-dimensional pose vector O of the end effector. pose (including position and quaternion information) to form a 15-dimensional feature vector (O pose ,O visual This composite feature vector is then fed into the controller network, which generates the action vector a output , guiding the end effector to perform the next operation. Action vector a output That is, the controller network predicts the predicted value of the pose data of the contact point at the next time step, and its training label a data The true value of the pose data of the contact point at the next time step in the projection data of the reference trajectory projected in the simulation environment.

[0114] Optionally, the mean square error loss function (MSE) can be used to measure the output action vector a output and its training label a dataThis loss function can quantify the accuracy of action prediction and provide guidance for controller network training. At the same time, the Adam optimizer is used to optimize the parameters of the visual encoder and controller network. The Adam optimizer is widely used due to its adaptive learning rate feature. It can automatically adjust the learning rate according to the historical information of the gradient during training, thereby accelerating convergence and improving training results.

[0115] After completing the imitation training in step S301 (where the visual encoder and controller are trained separately), a first control strategy model that can barely complete the basic target task has been obtained. According to the method 30 described in the previous section, the reinforcement learning training process in step S302 can be started to obtain a second control strategy model with stronger generalization ability.

[0116] Optionally, in operation S302, the process of determining the reward score for the control information of the end effector at the current time step includes: using a value network to determine the cumulative reward score for the control information of the end effector at the current time step. Specifically, the training process of reinforcement learning needs to introduce a value network. The value network receives the same input as the controller network and outputs a value for evaluating the action output generated by the controller network (that is, the "reward score corresponding to the learning task corresponding to the action round and the reward score related to the physical constraints in the process of simulating the execution of the learning task corresponding to the action round" mentioned above).

[0117] Therefore, the reinforcement learning training process can be modeled as a tuple Defined as a Markov decision process, where represents the state space, represents the action space, p(s ′ |s,a) is the state transition function, which means that after executing action a in state s, we reach state s. ′ The probability of . Function The cumulative reward score in the target task is defined, and its value is determined by the value network. γ∈[0,1) is a discount factor used to evaluate the importance of future reward scores.

[0118] As described above, in some embodiments of the present disclosure, the training process of reinforcement learning can also be combined with curriculum learning, so that the first control strategy model can not only continuously explore the environment of the tool operation task, but also continuously adjust and optimize the predicted action according to the exploration results. In this way, the first control strategy model can gradually improve the accuracy of the action to better adapt to and complete various tool operation tasks.

[0119] In the process of reinforcement learning, a sparse reward score "a reward score corresponding to the learning task corresponding to the action round and a reward score related to the physical constraints in the process of simulating the execution of the learning task corresponding to the action round" is used to describe the task requirements, wherein "a reward score corresponding to the learning task corresponding to the action round" is only given when the task is completed. In addition, the embodiment of the present disclosure also designs "a reward score related to the physical constraints in the process of simulating the execution of the learning task corresponding to the action round" to constrain the action to meet the physical constraints of the environment and prevent undesirable training results.

[0120] Specifically, the reward score can be defined in the following way to guide the end effector to perform the learning task in a desired way.

[0121] Regarding the "reward score corresponding to the learning task corresponding to the completed action round", if the virtual body corresponding to the end effector successfully completes the task, that is, successfully scoops up a target object, the reward score will increase by 150 points; and for each target object that overflows, the reward score will be subtracted by 250 points. Therefore, such a reward score will guide the end effector to learn how to scoop objects effectively while avoiding too many objects from being scattered, ensuring that its behavior meets the requirements of accuracy and efficiency in actual operations.

[0122] Regarding the "bonus points related to physical constraints during the simulation of the learning task corresponding to the action round", if the target tool (such as a spoon) collides with a bowl or object, the target tool falls off the target actuator, or the target tool exceeds the preset position limit, the bonus points will be deducted by 100 points. These bonus points are intended to ensure that the end effector keeps the target tool stable during operation, avoids collisions, and limits its movements within the workspace of the end effector. In this way, the end effector can learn how to complete tasks efficiently while complying with physical constraints.

[0123] In the course design of the tool manipulation task, the emphasis is placed on the properties of the virtual body corresponding to the target tool, the properties of the virtual body corresponding to the target object, and the properties of the task environment (e.g., the virtual body of the container in which the target object is located). Taking the scooping task as an example, the course difficulty can be achieved by adjusting three key factors: the height of the target object in the container, the size of the target object, and the size of the target tool - the spoon.

[0124] Alternatively, when the height of the target object in the container is lower, the spoon needs to reach deeper to touch the object, which not only increases the risk of the spoon colliding with the container wall and bottom, but also makes the entire scooping task more challenging. Similarly, the larger the size of the target object, the smaller the amount of object that the spoon can scoop in a single operation, which also increases the difficulty of the course accordingly. In addition, using a smaller spoon will limit the range of visual perception, making it more difficult for the visual encoder to capture object features, further increasing the complexity of the task. Through these meticulous adjustments, the course can gradually improve the students' operational skills and adaptability.

[0125] The present disclosure designs multiple structured courses that mix different target object heights, target object sizes, and target tool sizes as multidimensional difficulty variables for the control policy model to learn various scooping tasks during reinforcement learning training. The difficulty information of these structured courses is evaluated and sorted from easy to difficult, allowing the control policy model to steadily improve performance and acquire new skills during learning. By continuously evaluating the performance of the control policy model in the current reinforcement learning process, it can be decided whether to switch to a more difficult or easier course. In addition, previously learned courses are interspersed at certain stages to prevent potential problems of catastrophic forgetting of the control policy network. This course design strategy is intended to promote the robustness of the control policy model by gradually increasing the difficulty and reviewing regularly.

[0126] For example, in one embodiment of the present disclosure, the course can be set up in two stages, and then the learning tasks corresponding to each action round can be designed. Of course, the present disclosure is not limited to this.

[0127] Specifically, in the first stage, we can first determine the courses The difficulty variable types in the task are as follows, such as the size of the target tool, the size of the target object, the number of target objects, the initial position of the target tool, the target position of the target tool, the container in which the target object is located, etc. In the scooping task, the difficulty variable types are as follows, such as the height of the target object in the container, the size of the target object, and the size of the target tool-the spoon, and the difficulty coefficient corresponding to each specific difficulty variable is evaluated, such as 1 cm or small size. Then, different types of difficulty variables are combined to obtain different learning tasks, and the combined difficulty coefficients of these learning tasks are calculated. Next, these courses are sorted in ascending order according to the course difficulty information.

[0128] In the second phase, the strategy model is trained according to this structured curriculum and the control strategy model is continuously evaluated in the current curriculum. If the control strategy model is in the current course The performance of the pass, then switch to the next more difficult course Training; if the model is in the current course The performance of is lower than the preset failure threshold δ failed , or if the passing threshold is still not reached within the maximum number of training cycles n, it will switch to the previous simpler course In addition, if the model can be continuously upgraded to reach the review threshold e times, a review course will be interspersed This design aims to ensure that the control strategy model will gradually transition to higher-difficulty courses after mastering the skills at the current difficulty level, while consolidating the learned knowledge through review to prevent forgetting.

[0129] Specifically, you can first set the course set For course collections Each course Determine the corresponding course difficulty coefficient V i Course Difficulty V i Can be composed of multiple dimensions, such as {v1,v2,...,v m}. Then based on the equation Calculate course difficulty information C i , which is the difficulty coefficient measurement function d(v k ) in each dimension. Then, all courses can be sorted according to the course difficulty information C through the function D=Sort(D,C).

[0130] Then, the control strategy model is trained starting from the simplest course and looping through the learning tasks in all sorted courses. The training is performed in , and the model parameters and reward scores of the control strategy model are adjusted accordingly. If the value of the performance measurement function P(M, reward) is greater than or equal to the threshold δ pass If you pass e courses in a row, review the first t courses If you fail to pass e courses consecutively, start studying the next course.

[0131] If the performance of the control strategy model in the current course is lower than the preset failure threshold δ failed , it will fall back to the previous course to continue training until the upgrade conditions are met. If the passing threshold δ is not reached within the maximum number of training cycles n pass , then it will switch to the previous simpler course Conduct training.

[0132] Finally, after the control strategy model passes all the challenges of the courses, it outputs the adjusted first control strategy model as the second control strategy model. This training method of gradually increasing difficulty helps the model gradually master more complex skills and eventually achieve optimal performance. Through regular review and timely adjustment of training difficulty, this course learning strategy can effectively improve the generalization ability and adaptability of the control strategy model.

[0133] After completing the training process of reinforcement learning, the adjusted first control strategy model can also be tested. The control strategy model is tested on each course, and the evaluation result of the performance measurement function P is compared with the threshold δ pass The comparison determines whether the control policy model should move on to the next more difficult course (if performance exceeds a threshold) or return to the previous easier course for additional training (if performance falls below a threshold). This process continues until the model can successfully pass the test of all environments, gradually improving its performance at different difficulty levels.

[0134] In addition, since the controller network already has a certain ability after the pre-training stage, directly optimizing the controller network and value network with mismatched capabilities may cause failures or longer training times in the early stages of training due to the inability of the untrained value network to accurately evaluate the output of the controller network. Therefore, in the initial stages of reinforcement learning training, the parameters of the controller network will be frozen first, focusing on training the value network. Once this value network has sufficient evaluation capabilities, the parameters of both networks will be unfrozen and trained simultaneously. The training process can be briefly described as follows.

[0135] Optionally, in the initial stage of reinforcement learning training, only the value network can be trained. The training of the value network includes: performing multiple action rounds for the first target task, and in each time step in each action round: based on the scene image at the current time step during the simulation of the execution of the first target task by the end effector and the posture corresponding to the contact point between the end effector and the first target tool, using the controller network in the first control strategy model, determine the control information of the end effector at the current time step; and based on the control information of the end effector at the current time step, predict the predicted value of the cumulative reward score of the control information of the end effector at the current time step; after completing each action round, calculate the predicted value of the cumulative reward score of the control information of the last time step in the action round as the predicted value of the reward corresponding to the action round; based on the execution result of the corresponding first target task in the action round, determine the true value of the reward score corresponding to the action round; and adjust the model parameters in the value network so that the predicted value of the reward corresponding to the action round and the difference between the predicted value converge. Of course, the present disclosure is not limited to this.

[0136] Optionally, in a subsequent stage of reinforcement learning training, both the value network and the controller network may be jointly trained. The joint training of the value network and the controller network includes: performing multiple action rounds for the second target task, in each time step in each action round: in each time step in each action round: based on the scene image at the current time step during the process of the end effector performing the learning task and the posture corresponding to the contact point between the end effector and the second target tool, using the controller network of the first control strategy model in training, determining the predicted value of the control information of the end effector at the current time step; based at least in part on the predicted value of the control information of the end effector at the current time step, using the value network in training, determining the predicted value of the control information of the end effector at the current time step; The method comprises the steps of: calculating a predicted value of the cumulative reward score of the control information of the actuator at the current time step; calculating, after completing each action round, a predicted value of the cumulative reward score of the control information of the last time step in the action round as a predicted value of the reward score corresponding to the action round; determining a true value of the reward score corresponding to the action round based on the execution result of the corresponding learning task in the action round; adjusting the model parameters in the value network so that the difference between the predicted value and the true value of the reward score corresponding to the action round converges; and adjusting the model parameters of the controller network in the first control strategy model so that the predicted value of the reward score corresponding to the action round increases.

[0137] Optionally, in step S302, reinforcement learning training can be performed in a simulation environment. Optionally, a depth image of the simulation environment is captured using the built-in camera of MuJoCo and input into the visual encoder trained in step S302. The posture of the virtual spoon is obtained directly from MuJoCo. A reinforcement learning training environment is constructed using OpenAI Gym. The observation space is defined as a 15-dimensional vector in the range of (-∞, +∞), and the action space is a 6-dimensional vector in the range of [-1, 1]. The execution time corresponding to the control information of the end effector at the current time step can be a fixed duration, such as 0.002 seconds, until a round of action is completed. At each time step, the simulation environment receives a 6-dimensional vector a output , which represents the control information output by the controller network in training and is used to control the movement of the virtual spoon. Subsequently, the simulation environment returns the updated depth image and the latest pose information of the virtual contact point. Therefore, this setting can accurately simulate the dynamic process of the scooping task while providing a structured observation and action space for the reinforcement learning algorithm. By updating the state of the simulation environment and the pose information of the contact point in real time, the continuous scooping action can be effectively simulated, providing rich feedback information for adjusting the first control strategy model.

[0138] Next, if Figure 5As shown, after completing step S302, the second control strategy model can be directly deployed on the real machine of the end effector to perform one or more target tasks, especially the first target task or the second target task.

[0139] At this time, the input of the second control strategy model may be a scene image captured in real time in the physical space, and the output of the second control strategy model may be control information in the physical space for controlling the end effector.

[0140] Optionally, the control information is the expected posture information of the contact point between the end effector and the second target tool in the next time step. Optionally, after determining the expected posture information of the contact point between the end effector and the second target tool in the next time step, the control information of each motor of the end effector can be determined by the inverse kinematics method, which can be either the acceleration of each motor or the torque of the motor. Although in mathematical terms, there is not much difference between these two physical quantities as control information for controlling the rotation of the motor, in actual physical systems, not both of these physical quantities can be accurately measured. Therefore, in the actual use of the end effector, those skilled in the art can, according to the specific circumstances, select physical quantities with better data test results and more consistent with the model for subsequent calculations. Of course, the present disclosure is not limited to this.

[0141] For example, inverse kinematics is a method for calculating the rotation angle and speed of each component of the end effector based on the desired posture information of the end effector (e.g., the desired posture information of the contact point between the end effector and the target tool). Using inverse kinematics, the target motion control of the end effector can be achieved.

[0142] The process of determining the control information required for the end effector to perform the second target task by the inverse kinematics method can be mathematically described by the following formula (1).

[0143] q=IK(p end -p base ) (1)

[0144] Where q is the angle value vector of each joint of the end effector in the physical space, p end -p base is the position vector of the contact point between the end effector and the second target tool. Formula (1) establishes a mapping from the space corresponding to each joint to the Cartesian space. For example, a numerical method or an analytical method can be used to solve the joint angle that satisfies the target position and posture based on the dynamic model corresponding to the end effector, and then the joint angle information is used to determine the control information for controlling each joint so that the end effector can perform the target action.

[0145] Optionally, when there is no analytical solution for the inverse kinematics mapping, a numerical approximation method can be used. Given the position vector p of the end effector in physical space, end -p base , the joint angle solution q that satisfies the constraints is searched through an iterative algorithm. After the inverse kinematics solution is obtained, the kinematic solution q in the joint space corresponds to the end effector reaching the target posture p end The joint control amount can be obtained by further dynamic analysis, calculating the joint driving force / torque, completing motion control, and obtaining control information in the physical space. Of course, the present disclosure is not limited to this.

[0146] Therefore, the embodiment of the present disclosure further provides an end effector, comprising: a central connecting member and a plurality of multi-joint operating components; a control strategy model, the control strategy model being carried on the end effector and determined by the method 30 according to the embodiment of the present disclosure. The control strategy model may be the first control strategy model or the second control strategy model mentioned above, and the present disclosure is not limited thereto.

[0147] Optionally, the end effector is configured to perform a second target task, and the performing of the second target task includes: at the current moment, obtaining a reference scene image when the end effector performs the second target task and a posture corresponding to the contact point between the end effector and the second target tool, using a second control strategy model carried on the end effector to generate control information for controlling the end effector at the current time step, and based on the control information for controlling the end effector at the current time step, determining the motor drive information of the central connecting member and the multiple multi-joint operating components; and controlling the end effector based on the motor drive information of the central connecting member and the multiple multi-joint operating components.

[0148] Optionally, the end effector may also be configured to perform a first target task. In this case, although the first control strategy model may also be used to complete the task, generally speaking, the control effect of the second control strategy model will be better than that of the first control strategy model. Therefore, the process of performing the first target task may include: at the current moment, obtaining a reference scene image when the end effector performs the first target task and the posture corresponding to the contact point between the end effector and the second target tool, using the second control strategy model carried on the end effector, generating control information for controlling the end effector at the current time step, based on the control information for controlling the end effector at the current time step, determining the motor drive information of the central connector and the multiple multi-joint operating components; and controlling the end effector based on the motor drive information of the central connector and the multiple multi-joint operating components.

[0149] Next, refer to Figure 6 The method 60 for controlling the end effector according to the embodiment of the present disclosure is further described. Figure 6 is a flow chart illustrating a method 60 for controlling an end effector according to an embodiment of the present disclosure.

[0150] The method 60 for controlling the end effector according to an embodiment of the present disclosure may include: Figure 5 The steps S601 to S603 are shown. Of course, the method 60 may also include more or fewer operations, and the present disclosure is not limited thereto.

[0151] In step S601, at the current moment, a reference scene image when the end effector performs a target task and a posture corresponding to a contact point between the end effector and a target tool are acquired.

[0152] In step S602, control information for controlling the end effector at the current time step is generated using the control strategy model carried by the end effector.

[0153] In step S603, based on the control information for controlling the end effector at the current time step, the motor drive information of the central connecting member and the plurality of multi-joint operating components is determined.

[0154] In step S604, the end effector is controlled based on the motor driving information of the central connecting member and the plurality of multi-joint operating assemblies.

[0155] Wherein, the control strategy model is the second control strategy model described above, which is obtained by referring to Figure 3 to Figure 4 The method described is determined.

[0156] Figure 7 A schematic diagram of a device 70 for controlling an end effector according to some embodiments of the present disclosure is schematically shown. The end effector comprises a central connecting piece and a plurality of multi-joint operating components.

[0157] like Figure 7As shown, the device 70 includes: a camera 71, which is used to obtain a reference scene image when the end effector performs a target task at the current moment; a sensor 72, which is used to determine the posture corresponding to the contact point between the end effector and the second target tool at the current moment; a processor 73, which is used to generate control information for controlling the end effector at the current time step using the control strategy model carried by the end effector; and based on the control information for controlling the end effector at the current time step, determine the motor drive information of the central connecting member and the multiple multi-joint operating components; and a motor 74, which is used to control the end effector based on the motor drive information of the central connecting member and the multiple multi-joint operating components. Wherein, the control strategy model is composed of a reference Figure 3 to Figure 4 The method described is determined.

[0158] Figure 8 An exemplary block diagram of an apparatus 80 for determining a control strategy model according to some embodiments of the present disclosure is schematically shown. The control strategy model is used to control an end effector, which includes a central connecting piece and a plurality of multi-joint operating components.

[0159] Specifically, the device 80 includes: a first module 81, used to obtain a first control strategy model carried by the end effector; a second module 82, used to adjust the first control strategy model carried by the end effector using multiple action rounds to determine the second control strategy model carried by the end effector, wherein, in each action round of the multiple action rounds, the learning task corresponding to the action round is determined based on the information of the action round; the virtual body corresponding to the end effector simulates the execution of the learning task corresponding to the action round using the first control strategy model being adjusted, and determines the reward score corresponding to the completion of the learning task corresponding to the action round and the reward score related to the physical constraints in the process of simulating the execution of the learning task corresponding to the action round; and adjusts the first control strategy model so that the reward score for completing the learning task corresponding to the action round is increased.

[0160] Fig. 9 An exemplary block diagram of a device 90 for controlling an end effector according to some embodiments of the present disclosure is schematically shown. The end effector includes a central connecting member and a plurality of multi-joint operating assemblies. The device 90 includes a first module 91, a second module 92, a third module 93 and a fourth module 94.

[0161] Specifically, the first module 91 is used to obtain, at the current moment, a reference scene image when the end effector performs a target task and the posture corresponding to the contact point between the end effector and the target tool. The second module 92 is used to generate control information for controlling the end effector at the current time step using the control strategy model carried by the end effector. The third module 93 is used to determine the motor drive information of the central connecting member and the multiple multi-joint operating components based on the control information for controlling the end effector at the current time step. The fourth module 94 is used to control the end effector based on the motor drive information of the central connecting member and the multiple multi-joint operating components.

[0162] Wherein, the control strategy model is the second control strategy model described above, which is obtained by referring to Figure 3 to Figure 4 The method described is determined.

[0163] It should be understood that the devices 80 and 90 can be implemented in software, hardware or a combination of software and hardware. Multiple different modules can be implemented in the same software or hardware structure, or one module can be implemented by multiple different software or hardware structures.

[0164] In addition, the device 80 can be used to implement the method 30 described above, and the device 90 can be used to implement the method 60 described above. The relevant details have been described in detail above, and for the sake of brevity, they will not be repeated here. The devices 80 and 90 can have the same features and advantages as described in relation to the aforementioned methods.

[0165] Fig.10 An example block diagram of a computing device 1900 according to some embodiments of the present disclosure is schematically shown. For example, it may represent a computing device that may be used to deploy the apparatus 80 and the apparatus 90 provided by the present disclosure.

[0166] As shown, the example computing device 1900 includes a processing system 1901, one or more computer-readable media 1902, and one or more I / O interfaces 1903 that are communicatively coupled to each other. Although not shown, the computing device 1900 may also include a system bus or other data and command transmission system that couples the various components to each other. The system bus may include any one or combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus utilizing any of a variety of bus architectures, or may also include such as control and data lines.

[0167] Processing system 1901 represents the functionality of performing one or more operations using hardware. Thus, processing system 1901 is illustrated as including hardware elements 1904 that may be configured as processors, functional blocks, and the like. This may include implementing application specific integrated circuits in hardware or other logic devices formed using one or more semiconductors. Hardware elements 1904 are not limited by the materials from which they are formed or the processing mechanisms employed therein. For example, a processor may be composed of (multiple) semiconductors and / or transistors (e.g., electronic integrated circuits (ICs)). In such a context, processor executable instructions may be electronic executable instructions.

[0168] Computer-readable medium 1902 is illustrated as including memory / storage 1905. Memory / storage 1905 represents a memory / storage associated with one or more computer-readable media. Memory / storage 1905 may include volatile storage media (such as random access memory (RAM)) and / or non-volatile storage media (such as read-only memory (ROM), flash memory, optical disk, magnetic disk, etc.). Memory / storage 1905 may include fixed media (e.g., RAM, ROM, fixed hard drive, etc.) and removable media (e.g., flash memory, removable hard drive, optical disk, etc.). Exemplarily, memory / storage 1905 may be used to store the various poses, position data, etc. mentioned in the above embodiments. Computer-readable medium 1902 may be configured in various other ways further described below.

[0169] One or more input / output interfaces 1903 represent functionality that allows a user to enter commands and information into the computing device 1900 and also allows information to be presented to the user and / or sent to other components or devices using various input / output devices. Examples of input devices include a keyboard, a cursor control device (e.g., a mouse), a microphone (e.g., for voice input), a scanner, touch functionality (e.g., a capacitive or other sensor configured to detect physical touch), a camera (e.g., that can detect motion that does not involve touch as gestures using visible or invisible wavelengths (such as infrared frequencies)), a network card, a receiver, and the like. Examples of output devices include a display device (e.g., a monitor or projector), a speaker, a printer, a tactile response device, a network card, a transmitter, and the like.

[0170] The computing device 1900 also includes an application 1906. The application 1906 can be stored as a computer program instruction in the memory / storage device 1905. The application 1906 can be implemented together with the processing system 1901 and the like. Figure 7 or Figure 8 The overall functions of the various modules of the device 70 or 80 are described.

[0171] Various techniques may be described herein in the general context of software, hardware, elements, or program modules. Typically, these modules include routines, programs, objects, elements, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The terms "module," "function," etc., as used herein, generally represent software, firmware, hardware, or a combination thereof. The features of the techniques described herein are platform-independent, meaning that these techniques can be implemented on a variety of computing platforms with a variety of processors.

[0172] An implementation of the described modules and techniques may be stored on or transmitted across some form of computer-readable media. Computer-readable media may include various media accessible by computing device 1900. By way of example and not limitation, computer-readable media may include “computer-readable storage media” and “computer-readable signal media”.

[0173] In contrast to a simple signal transmission, carrier wave, or signal itself, "computer-readable storage medium" refers to a medium and / or device capable of persistently storing information, and / or a tangible storage device. Therefore, a computer-readable storage medium refers to a non-signal-bearing medium. Computer-readable storage media include hardware such as volatile and non-volatile, removable and non-removable media and / or storage devices implemented in a method or technology suitable for storing information (such as computer-executable instructions, data structures, program modules, logic elements / circuits, or other data). Examples of computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage device, hard disk, cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or other storage device, tangible medium, or article of manufacture suitable for storing desired information and accessible by a computer.

[0174] "Computer readable signal media" refers to signal bearing media that is configured to send instructions to the hardware of the computing device 1900, such as via a network. Signal media typically can embody computer executable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave, data signal or other transmission mechanism. Signal media also includes any information transfer media. By way of example and not limitation, signal media include wired media such as a wired network or direct connection and wireless media such as acoustic, RF, infrared and other wireless media.

[0175] As previously described, hardware elements 1904 and computer-readable media 1902 represent instructions, modules, programmable device logic, and / or fixed device logic implemented in hardware form, which in some embodiments can be used to implement at least some aspects of the technology described herein. Hardware elements can include integrated circuits or systems on a chip, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), and other implementations in silicon or components of other hardware devices. In this context, hardware elements can be used as processing devices for executing program tasks defined by instructions, modules, and / or logic embodied by hardware elements, as well as hardware devices for storing instructions for execution, such as the computer-readable storage media described previously.

[0176] The aforementioned combination may also be used to implement the various techniques and modules described herein. Therefore, software, hardware or program modules and other program modules may be implemented as one or more instructions and / or logic embodied on some form of computer-readable storage medium and / or by one or more hardware elements 1904. Computing device 1900 may be configured to implement specific instructions and / or functions corresponding to software and / or hardware modules. Therefore, for example, by using a computer-readable storage medium and / or hardware element 1904 of a processing system, a module may be implemented as a module executable by computing device 1900 as software, at least in part, in hardware. Instructions and / or functions may be executed / operable by, for example, one or more computing devices 1900 and / or processing system 1901 to implement the techniques, modules and examples described herein.

[0177] The techniques described herein may be supported by these various configurations of computing device 1900 and are not limited to the specific examples of the techniques described herein.

[0178] It should be understood that, for the sake of clarity, embodiments of the present disclosure are described with reference to different functional units. However, it will be apparent that, without departing from the present disclosure, the functionality of each functional unit can be implemented in a single unit, in multiple units, or as a part of other functional units. For example, the functionality that is described as being performed by a single unit can be performed by multiple different units. Therefore, reference to a specific functional unit is only considered as a reference to a suitable unit for providing the described functionality, rather than indicating a strict logical or physical structure or organization. Therefore, the present disclosure can be implemented in a single unit, or can be physically and functionally distributed between different units and circuits.

[0179] The present disclosure provides a computer-readable storage medium having computer-executable instructions stored thereon. When the computer-executable instructions are executed, the method for determining a control strategy model, the method for determining a tool control model, or the control method described above is implemented.

[0180] The present disclosure provides a computer program product or a computer program, which includes computer executable instructions, which are stored in a computer-readable storage medium. A processor of a computing device reads the computer executable instructions from the computer-readable storage medium, and the processor executes the computer executable instructions, so that the computing device executes the method for determining a control strategy model, the method for determining a tool control model, or the control method provided in the above various embodiments.

[0181] Variations to the disclosed embodiments will be understood and effected by those skilled in the art in practicing the claimed subject matter by studying the drawings, the disclosure and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and "a" or "an" does not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. A method for determining a control strategy model, wherein the control strategy model is used to control an end effector, wherein the end effector includes a central connecting member and a plurality of multi-joint operating components, the method comprising: Acquire a first control strategy model carried by the end effector, and The first control strategy model carried by the end effector is adjusted by using a plurality of action rounds to determine the second control strategy model carried by the end effector, Wherein, in each of the plurality of action rounds, Based on the information of the action round, determining the learning task corresponding to the action round; The virtual body corresponding to the end effector simulates and executes the learning task corresponding to the action round using the first control strategy model being adjusted, and determines a reward score corresponding to completing the learning task corresponding to the action round and a reward score related to the physical constraint in the process of simulating and executing the learning task corresponding to the action round; and The first control strategy model is adjusted so that the reward score for completing the learning task corresponding to the action round and the reward score related to the physical constraints in the process of simulating the execution of the learning task corresponding to the action round are increased.

2. The method of claim 1, wherein: The first control strategy model is used to control the end effector to perform a first target task, wherein, during the process of the end effector performing the first target task, the end effector uses a first target tool to move a first target object from the first initial position to the first target position; The second control strategy model carried on the end effector is used to control the end effector to perform a second target task, wherein, during the process of the end effector performing the second target task, the end effector uses the second target tool to move the second target object from the second initial position to the second target position, and the first target task and the second target task are different; In the process of the virtual body corresponding to the end effector simulating the execution of the learning task corresponding to the action round, the end effector simulates the execution of: using the target tool specific to the action round to move the target object specific to the action round from the initial position specific to the action round to the target position corresponding to the action round.

3. The method of claim 2, wherein: The reward score for completing the learning task corresponding to the action round indicates at least one or a combination of the following items: whether the end effector successfully moves the target object specific to the action round from the initial position specific to the action round to the target position corresponding to the action round using the target tool specific to the action round; or Whether the target object overflows during the process in which the virtual body corresponding to the end effector simulates and performs the learning task corresponding to the action round.

4. The method of claim 2, wherein: The reward score related to the physical constraints during the process of simulating the execution of the learning task corresponding to the action round indicates at least one or a combination of the following items: During the process in which the virtual body corresponding to the end effector simulates and performs the learning task corresponding to the action round, whether the target tool specified in the action round collides with the target object specified in the action round; During the process in which the virtual body corresponding to the end effector simulates and performs the learning task corresponding to the action round, whether the target tool specific to the action round collides with the container where the target object specific to the action round is located; During the process in which the virtual body corresponding to the end effector simulates and performs the learning task corresponding to the action round, whether the end effector is separated from the target tool; as well as During the process in which the virtual body corresponding to the end effector simulates and performs the learning task corresponding to the action round, whether the target tool is beyond the observation range.

5. The method of claim 2, wherein: The determining, based on the information of the action round, the learning task corresponding to the action round comprises: Based on the information of the action round including the course difficulty information and the course progress information corresponding to the action round, a target tool specific to the action round or a target object specific to the action round is determined.

6. The method of claim 5, wherein: The course difficulty information corresponding to the current action round is associated with the course progress information of the current action round, and the course progress information indicates that the later the action round is executed, the higher the course difficulty information corresponding to the current action round is.

7. The method of claim 6, wherein: The same course difficulty information corresponds to multiple action rounds, and different action rounds correspond to different learning tasks. The learning tasks corresponding to different action rounds satisfy at least one of the following conditions: The specific target tools are different for different action rounds; The specific target objects of different action rounds are different; The number of specific target objects varies between action rounds; or Target objects specific to different action rounds are located in different containers.

8. The method of claim 2, wherein: The adjusting the first control strategy model carried by the end effector by using multiple action rounds to determine the second control strategy model carried by the end effector includes: In each of the plurality of action rounds, In the process of simulating the execution of the learning task by the end effector, based on the scene image at the current time step and the pose information corresponding to the contact point between the end effector and the target tool specified in the action round, the control information of the end effector at the current time step is determined by using the first control strategy model being adjusted; Determining a reward score for the control information of the end effector at the current time step based at least in part on the control information of the end effector at the current time step; and After completing the action round, adjusting the first control strategy model carried by the end effector based at least in part on the reward score for the control information of the end effector at each time step.

9. The method of claim 8, wherein: The determining of the reward score for the control information of the end effector at the current time step includes: using a value network to determine the cumulative reward score for the control information of the end effector at the current time step, The training of the value network at least includes: performing multiple action rounds for the first target task, and in each time step in each action round: Determining control information of the end effector at the current time step using the controller network in the first control strategy model based on the scene image at the current time step during the process of the end effector simulating the execution of the first target task and the pose corresponding to the contact point between the end effector and the first target tool; and Based on the control information of the end effector at the current time step, predict a predicted value of a cumulative reward score of the control information of the end effector at the current time step; After completing each round of actions, Calculate the predicted value of the cumulative reward score of the control information of the last time step in the action round as the predicted value of the reward corresponding to the action round; Determining a true value of a reward score corresponding to the action round based on an execution result of the first target task corresponding to the action round; and The model parameters in the value network are adjusted so that the predicted value of the reward corresponding to the action round and the difference between the predicted value converge.

10. The method of claim 9, wherein: The adjusting the first control strategy model carried by the end effector by using multiple action rounds to determine the second control strategy model carried by the end effector includes: At each time step in each action round: Determine a predicted value of control information of the end effector at the current time step based on a scene image at the current time step during the end effector performing the learning task and a pose corresponding to a contact point between the end effector and a second target tool, using a controller network of a first control strategy model in training; Determining, using the value network in training, a predicted value of a cumulative reward score for the control information of the end effector at the current time step based at least in part on the predicted value of the control information of the end effector at the current time step; After completing each round of actions, Calculate the predicted value of the cumulative reward score of the control information of the last time step in the action round as the predicted value of the reward score corresponding to the action round; Determining a true value of a reward score corresponding to the action round based on an execution result of the learning task corresponding to the action round; Adjusting the model parameters in the value network so that the difference between the predicted value and the actual value of the reward score corresponding to the action round converges; and Adjust the model parameters of the controller network in the first control strategy model so that the predicted value of the reward score corresponding to the action round increases.

11. The method of claim 2, wherein: During the process of the first target object moving from the first initial position to the first target position, the first target tool provides a force to the first target object, the end effector does not contact the first target object, and During the process of the second target object moving from the second initial position to the second target position, the second target tool provides a force to the second target object, and the end effector is not in contact with the second target object.

12. The method of claim 1, wherein: The first target task and the second target task do not satisfy at least one of the following conditions: The first target tool is different from the second target tool; The first target object is different from the second target object; The number of the first target objects is different from the number of the second target objects; The first initial position is different from the second initial position; The first target location is different from the second target location; or The first target object is located in a first container, the second target object is located in a second container, and the first container and the second container are different.

13. A method for controlling an end effector, the end effector comprising a central connecting member and a plurality of multi-joint operating assemblies, comprising: At the current moment, a reference scene image when the end effector performs a target task and a posture corresponding to a contact point between the end effector and a target tool are obtained; Using the control strategy model carried by the end effector, generating control information for controlling the end effector at a current time step; Determining motor drive information of the central connecting member and the plurality of multi-joint operating components based on the control information for controlling the end effector at the current time step; as well as Controlling the end effector based on the motor drive information of the central connecting member and the plurality of multi-joint operating components; Wherein, the control strategy model is determined by the method according to any one of claims 1-11.

14. A device for controlling an end effector, the end effector comprising a central connecting member and a plurality of multi-joint operating assemblies, the device comprising: A camera, used to obtain a reference scene image when the end effector performs a target task at a current moment; A sensor for determining the posture corresponding to the contact point between the end effector and the target tool at the current moment; A processor, configured to generate control information for controlling the end effector at a current time step by using a control strategy model carried by the end effector; and to determine motor drive information of the central connecting member and the plurality of multi-joint operating components based on the control information for controlling the end effector at the current time step; as well as a motor for controlling the end effector based on motor drive information of the central connecting member and the plurality of multi-joint operating assemblies; Wherein, the control strategy model is determined by the method according to any one of claims 1-11.

15. A device for determining a control strategy model, wherein the control strategy model is used to control an end effector, wherein the end effector comprises a central connecting member and a plurality of multi-joint operating components, the device comprising: A first module is used to obtain a first control strategy model carried by the end effector; The second module is used to adjust the first control strategy model carried by the end effector by using multiple action rounds to determine the second control strategy model carried by the end effector, Wherein, in each of the plurality of action rounds, Based on the information of the action round, determining the learning task corresponding to the action round; The virtual body corresponding to the end effector simulates and executes the learning task corresponding to the action round using the first control strategy model being adjusted, and determines a reward score corresponding to completing the learning task corresponding to the action round and a reward score related to the physical constraint in the process of simulating and executing the learning task corresponding to the action round; and The first control strategy model is adjusted so that the reward score for completing the learning task corresponding to the action round is increased.

16. A device for controlling an end effector, the end effector comprising a central connecting member and a plurality of multi-joint operating assemblies, comprising: The first module is used to obtain, at the current moment, a reference scene image when the end effector performs a target task and a posture corresponding to a contact point between the end effector and a target tool; The second module is used to generate control information for controlling the end effector at a current time step by using a control strategy model carried by the end effector; A third module is used to determine the motor driving information of the central connecting member and the plurality of multi-joint operating components based on the control information for controlling the end effector at the current time step; as well as A fourth module, for controlling the end effector based on the motor drive information of the central connecting member and the plurality of multi-joint operating components; Wherein, the control strategy model is determined by the method according to any one of claims 1-11.

17. An electronic device comprising: processor; and A memory, wherein the memory stores a computer executable code, and when the computer executable code is executed by the processor, the method according to any one of claims 1 to 13 is executed.

18. A non-volatile computer-readable storage medium having executable codes stored thereon, wherein when the executable codes are executed by a processor, the processor is caused to execute the method according to any one of claims 1 to 13.

19. A computer program product comprising computer executable instructions, which implement the method of any one of claims 1 to 13 when executed by a processor.

Images