A training method and a control method of a mechanical arm control model, a device, and a medium

Abstract

The application provides a mechanical arm control model training method, a control method, a device and electronic equipment. The training method comprises the following steps: storing a sample generated by each interaction between a pre-constructed simulation training arm and a simulation training environment in a pre-constructed experience pool module; calculating a second reward value of an interaction action in the sample according to state information of a simulation mechanical arm before and after the interaction and the interaction action through an action evaluation network module; obtaining at least one sample from the experience pool module randomly through a critic network, processing the sample to obtain a processing result, and determining a target action from a pre-configured action set according to the processing result through an actor network, so as to control the simulation mechanical arm to interact with the training environment by using the target action, generate a new sample and store the new sample in the experience pool module; training and optimizing the critic network and the actor network to obtain a trained control model, and realizing autonomous control of the mechanical arm.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to a training method, control method, device, and medium for a robotic arm control model. Background Technology

[0002] Currently, with the increasing variety of robotic arm types, they are becoming popular in many fields, such as medicine, manufacturing, and education and research. Robotic arms are categorized into different types based on their structural form, each with different degrees of freedom. By pre-inputting commands, they can perform tedious and repetitive tasks, freeing up human labor and significantly improving productivity. Furthermore, the commands are precise, resulting in higher quality execution compared to human labor. Due to their flexibility, robotic arms can also serve as a service product, assisting people with disabilities in performing simple tasks such as picking up objects and transporting items indoors, bringing convenience to their lives through simple tasks.

[0003] However, traditional robotic arms are controlled via commands, requiring different commands to be designed for each application scenario to achieve the desired goal, resulting in low versatility. To meet precision requirements, the design must consider multiple factors, including the robotic arm's degrees of freedom and the tasks it performs, demanding a high level of expertise from the designer. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a training method, control method, device and medium for a robotic arm control model. Through the trained control model, the robotic arm can achieve autonomous control, master the optimal control strategy through trial and error learning, reduce the difficulty of manual design, and quickly adapt to new tasks in similar scenarios with a small amount of learning, thereby improving the generalization ability of the robotic arm.

[0005] This application provides a method for training a robotic arm control model, comprising:

[0006] The samples generated from each interaction between the pre-built simulation training arm and the simulation training environment are stored in the pre-built experience pool module. The samples include the state information of the simulation robotic arm before the interaction, the interaction actions taken when interacting with the training environment, the state information of the simulation robotic arm after the interaction, and the first reward value fed back by the simulation training environment based on the interaction actions.

[0007] Based on the state information of the simulated robotic arm before interaction, the interactive actions taken when interacting with the training environment, and the state information of the simulated robotic arm after interaction in the sample, the second reward value of the interactive action in the sample is calculated through the action evaluation network module.

[0008] At least one sample is randomly obtained from the experience pool module through the commentator network, and the at least one sample and the second reward value corresponding to the sample are processed according to the preset processing rules to obtain the processing result. The processing result is output to the actor network. The actor network determines the target action and action value from the pre-configured action set according to the processing result, so as to control the simulated robotic arm to interact with the training environment by adopting the target action, generate a new sample and store it in the experience pool module.

[0009] An optimization target value is calculated based on the first reward value and the second reward value. The parameters of the commenter network are updated based on the optimization target value, and the parameters of the actor network are updated based on the action value, until the control model meets the training termination condition, thus obtaining a trained control model. The trained control model is used to control the target robotic arm to perform target actions.

[0010] In some embodiments, before storing the samples generated by each interaction between the pre-built simulation training arm and the simulation training environment in a pre-built experience pool module, the training method further includes:

[0011] The system collects attribute information of the target robotic arm and scene information of the target scene where the target robotic arm is located through an image acquisition device. Based on the attribute information of the target robotic arm and the scene information, it determines a set of state information, a set of actions, and reward rule information to construct a simulated robotic arm and a simulated training environment. The scene information includes the position of the target robotic arm and the target location information, target object information, and / or obstacle information in the target scene that interact with the target robotic arm.

[0012] In some embodiments, before storing the samples generated by each interaction between the pre-built simulation training arm and the simulation training environment in a pre-built experience pool module, the training method further includes:

[0013] A simulation training environment is constructed based on the structure of the target robotic arm and the target environment in which the target robotic arm is located; the simulation training environment includes the target location, target object and / or obstacle located at a preset position.

[0014] In some embodiments, in the training method of the robotic arm control model, the reward rule information is used to determine the first reward value of the interactive action based on the relative position information of the simulated robotic arm and the target location before and after the simulated robotic arm performs the interactive action, and the relative position information of the simulated robotic arm and the obstacle before and after the interaction.

[0015] In some embodiments, the training method for the robotic arm control model involves controlling the simulated robotic arm to interact with the training environment using the target action, generating a new sample, and storing it in the experience pool module; including:

[0016] The simulated robotic arm is controlled to interact with the training environment using the target action, and the state information of the simulated robotic arm before and after the interaction is obtained. Based on the state information of the simulated robotic arm before and after the interaction and the pre-configured reward rule information, the first reward value fed back by the simulated training environment based on the target action is determined.

[0017] Based on the state information of the simulated robotic arm before interaction, the target action, the state information of the simulated robotic arm after interaction, and the first reward value, a new sample is generated and stored in the experience pool module.

[0018] In some embodiments, updating the parameters of the commenter network based on the optimization target value includes:

[0019] Based on the optimization target value, the minimum loss function in the commenter network is calculated, and the gradient of the minimum loss function is calculated to determine the first target gradient value;

[0020] Update the parameters of the commentator network based on the first target gradient value;

[0021] Updating the parameters of the actor network based on the action value includes:

[0022] The gradient of the action values ​​output by the actor network is calculated to determine the gradient value of the second objective.

[0023] The parameters of the actor network are updated based on the second objective gradient value and the preset exploration bias term, and the preset exploration bias term is also updated based on the second objective gradient value.

[0024] In some embodiments, a control method for a robotic arm is also provided, the control method comprising:

[0025] Obtain the starting position information, destination position information, and target item information of the target robotic arm in the target scene;

[0026] Based on the starting location information, the target location information, and the target item information, a set of target actions for the target robotic arm is generated using a trained control model. The set of target actions includes actions by which the target robotic arm grasps or releases the target item, as well as actions by which it moves from the starting position to the target location. The trained control model is obtained by training based on the structure of the target robotic arm and the target scene.

[0027] The target robotic arm is controlled to execute each action in the target action set in sequence to transfer the target object to the target location.

[0028] In some embodiments, a training device for a robotic arm control model is also provided, the training device comprising:

[0029] The experience pool module is used to store samples generated by each interaction between the pre-built simulation training arm and the simulation training environment. The samples include the state information of the simulation robotic arm before the interaction, the interaction actions taken when interacting with the training environment, the state information of the simulation robotic arm after the interaction, and the first reward value fed back by the simulation training environment based on the interaction actions.

[0030] The action evaluation network module is used to calculate the second reward value of the interaction action in the sample based on the state information of the simulated robotic arm before interaction, the interaction action taken when interacting with the training environment, and the state information of the simulated robotic arm after interaction.

[0031] The action generation module is used to randomly obtain at least one sample from the experience pool module through the commentator network, process the at least one sample and the second reward value corresponding to the sample according to the preset processing rules to obtain the processing result, and output the processing result to the actor network. The actor network determines the target action from the pre-configured action set according to the processing result, so as to control the simulated robotic arm to interact with the training environment by adopting the target action, generate a new sample and store it in the experience pool module.

[0032] The training module is used to calculate the optimization target value based on the first reward value and the second reward value, update the parameters of the commenter network based on the optimization target value, update the parameters of the actor network based on the action value, until the control model meets the training termination condition, and obtain the trained control model. The trained control model is used to control the target robotic arm to perform the target action.

[0033] In some embodiments, an electronic device is also provided, the electronic device including: a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, the processor communicating with the memory via the bus when the electronic device is running, and the steps of the training method of the robotic arm control model or the control method of the robotic arm being executed by the processor when the machine-readable instructions are executed.

[0034] In some embodiments, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, performs the steps of the robotic arm training method or the robotic arm control method.

[0035] Based on this, this application provides a training method, training device, and medium for a robotic arm control model, as well as a robotic arm control method. The control model includes an experience pool module, samples randomly obtained from the experience pool module, and based on the interactive actions of these samples, as well as the state information of the simulated robotic arm before and after the interaction, a motion evaluation network module calculates a second reward value for the interactive actions in the sample. The control strategy of the robotic arm is continuously corrected using the first and second reward values ​​of each sample during each training process. The weights of the commentator network and the actor network are continuously optimized based on the processing results of the commentator network, thereby obtaining a trained control model. The control model is based on the structure of the target robotic arm and the location of the target robotic arm. The system autonomously generates a set of target actions in the target environment, enabling autonomous control of the robotic arm. This reduces the difficulty of manual design and improves the generalization performance of the robotic arm through intelligent control, allowing it to be applied in multiple similar scenarios and increasing the utilization rate of the same robotic arm. During training, after each action selection by the robotic arm, the environment provides a first reward value based on that action, and the action evaluation network module also provides a second reward value based on the action and the state before and after the action selection. The two are added together as reward information and fed back to the robotic arm. This ensures that even after consecutive incorrect action selections and when positive rewards are lacking in the environment, the robotic arm still has an action evaluation value to guide its learning, improving training efficiency and the accuracy of the target action set autonomously generated by the control model. Attached Figure Description

[0036] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 A flowchart illustrating the training method for the robotic arm control model according to an embodiment of this application is shown.

[0038] Figure 2 The flowchart of the method described in this application embodiment for controlling the simulated robotic arm to interact with the training environment using the target action, generating a new sample and storing it in the experience pool module is shown.

[0039] Figure 3 A flowchart illustrating the method for updating the parameters of the commenter network according to the optimization target value, as described in an embodiment of this application, is shown.

[0040] Figure 4 A flowchart illustrating the method for updating the parameters of an actor network based on the action value, as described in an embodiment of this application, is shown.

[0041] Figure 5 A flowchart of the robotic arm control method according to an embodiment of this application is shown;

[0042] Figure 6 A schematic diagram of the structure of the training device for the robotic arm control model described in an embodiment of this application is shown;

[0043] Figure 7 A schematic diagram of the structure of the electronic device described in an embodiment of this application is shown. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0045] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0046] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0047] Currently, with the increasing variety of robotic arm types, they are becoming popular in many fields, such as medicine, manufacturing, and education and research. Robotic arms are categorized into different types based on their structural form, each with different degrees of freedom. By pre-inputting commands, they can perform tedious and repetitive tasks, freeing up human labor and significantly improving productivity. Furthermore, the commands are precise, resulting in higher quality execution compared to human labor. Due to their flexibility, robotic arms can also serve as a service product, assisting people with disabilities in performing simple tasks such as picking up objects and transporting items indoors, bringing convenience to their lives through simple tasks.

[0048] However, traditional robotic arms are controlled via commands, requiring different commands to be designed for each application scenario to achieve the desired goal, resulting in low versatility. To meet precision requirements, the design must consider multiple factors, including the robotic arm's degrees of freedom and the tasks it performs, demanding a high level of expertise from the designer.

[0049] In recent years, deep reinforcement learning has achieved remarkable results in many fields. Through continuous interaction between the agent and the environment, the agent continuously corrects its current strategy based on the reward value fed back by the environment in each interaction, thereby achieving autonomous learning without human guidance.

[0050] Based on this, this application provides a training method for a robotic arm control model. The control model includes an experience pool module, samples randomly obtained from the experience pool module, and based on the interactive actions of these samples, as well as the state information of the simulated robotic arm before and after the interaction, a motion evaluation network module calculates a second reward value for the interactive actions in each sample. The control strategy of the robotic arm is continuously corrected using the first and second reward values ​​of each sample during each training process. The weights of the commentator network and the actor network are continuously optimized based on the processing results of the commentator network, thereby obtaining a trained control model. This control model is autonomously generated based on the structure of the target robotic arm and the target environment in which the target robotic arm is located. The target action set enables autonomous control of the robotic arm, reducing the difficulty of manual design. Furthermore, the intelligent control of the robotic arm improves its generalization performance, allowing it to be applied in multiple similar scenarios and increasing the utilization rate of the same robotic arm. During training, after each action selection by the robotic arm, the environment provides a first reward value based on that action, and the action evaluation network module also provides a second reward value based on the action and the states before and after the action selection. The two are added together as reward information fed back to the robotic arm. This ensures that even after consecutive incorrect action selections and when positive rewards are lacking in the environment, the robotic arm still has an action evaluation value to guide its learning, improving training efficiency and the accuracy of the target action set autonomously generated by the control model.

[0051] The training method for the robotic arm control model described in this application embodiment is applied to a robotic arm control model, which includes an experience pool module, a motion evaluation network module, and a motion generation module; the motion generation module includes a commentator network and an actor network; please refer to... Figure 1 , Figure 1 A flowchart of a method for training a robotic arm control model according to an embodiment of this application is shown. Specifically, the training method includes the following steps S101-S104.

[0052] S101. Store the samples generated by each interaction between the pre-built simulation training arm and the simulation training environment in the pre-built experience pool module; the samples include the state information of the simulation robotic arm before the interaction, the interaction actions taken when interacting with the training environment, the state information of the simulation robotic arm after the interaction, and the first reward value fed back by the simulation training environment based on the interaction actions.

[0053] S102. Based on the state information of the simulated robotic arm before interaction, the interactive actions taken when interacting with the training environment, and the state information of the simulated robotic arm after interaction in the sample, calculate the second reward value of the interactive action in the sample through the action evaluation network module.

[0054] S103. At least one sample is randomly obtained from the experience pool module through the commentator network, and the at least one sample and the second reward value corresponding to the sample are processed according to the preset processing rules to obtain the processing result. The processing result is output to the actor network. The actor network determines the target action and action value from the pre-configured action set according to the processing result, so as to control the simulated robotic arm to interact with the training environment by adopting the target action, generate a new sample and store it in the experience pool module.

[0055] S104. Calculate the optimization target value based on the first reward value and the second reward value, update the parameters of the commentator network based on the optimization target value, update the parameters of the actor network based on the action value, until the control model meets the training termination condition, and obtain the trained control model. The trained control model is used to control the target robotic arm to perform the target action.

[0056] In this embodiment, the control model pre-constructs a simulation training arm and a simulation training environment, and the motion set of the simulation robotic arm also needs to be pre-configured in step S103; that is, before executing step S101, the simulation robotic arm and simulation training environment must be constructed first, and the motion set and reward rule information of the simulation robotic arm must be pre-configured; based on this, before storing the samples generated by each interaction of the pre-constructed simulation training arm and simulation training environment in the pre-constructed experience pool module, the training method further includes:

[0057] The system collects attribute information of the target robotic arm and scene information of the target scene where the target robotic arm is located through an image acquisition device. Based on the attribute information of the target robotic arm and the scene information, it determines a set of state information, a set of actions, and reward rule information to construct a simulated robotic arm and a simulated training environment. The scene information includes the position of the target robotic arm and the target location information, target object information, and / or obstacle information in the target scene that interact with the target robotic arm.

[0058] The reward rule information is used to determine the first reward value of the interactive action based on the relative position information of the simulated robotic arm and the target location before and after the interactive action, and the relative position information of the simulated robotic arm and the obstacle before and after the interaction.

[0059] Here, before storing the samples generated from each interaction between the pre-built simulation training arm and the simulation training environment in the pre-built experience pool module, the training method further includes:

[0060] A simulation training environment is constructed based on the structure of the target robotic arm and the target environment in which the target robotic arm is located; the simulation training environment includes the target location, target object and / or obstacle located at a preset position.

[0061] In this embodiment, the attribute information of the target robotic arm includes its type and structure. The type of target robotic arm includes multi-joint robotic arms, Cartesian coordinate system robotic arms, and spherical coordinate system robotic arms. Different types of robotic arms are composed of multiple connected joints, such as the connection between a human upper arm, forearm, and wrist. The number of joints is the degree of freedom. The structure of the robotic arm includes the degrees of freedom and the connection relationships between the joints.

[0062] The scene information includes the position of the target robotic arm and the target location information, target object information and / or obstacle information of the target scene in which the target robotic arm interacts. The target location is the location that the target robotic arm wants to reach, the target object is the object that the robotic arm wants to grab or put down, and the obstacle is the obstacle encountered by the robotic arm in the process of moving from the starting point to the target location.

[0063] The basic elements in the learning process of a control model are: a set of states. Action set and reward rules information

[0064] The set of states The information in the image is determined based on different images of the current scene and is represented by three-dimensional coordinates (x, y, z). The relative positions of objects around the robotic arm can be determined by different images captured by the image acquisition device. For example, a robotic arm needs to determine the position of the object being transported, the position of the target location, and the position of obstacles along the way. Then, the relative positions of these objects are marked using three-dimensional coordinates with the position of the robotic arm itself as the origin.

[0065] The completion of each task by a robotic arm requires the coordination of different degrees of freedom. Just as a person needs the upper arm to move the forearm, and the forearm to move the wrist, these three elements work together to achieve the same result. Therefore, the set of three degrees of freedom is represented as (a1, a2, a3), where each dimension's movement primarily involves vertical, horizontal, and spatial movement. Here, the set of movements... That is, all the actions that the target robotic arm can perform. For example, the action set includes the following actions: moving 10cm forward in the horizontal direction (10cm, 0, 0), moving 20cm forward in the horizontal direction (20cm, 0, 0), moving 20cm forward in the vertical direction (20cm, 0, 0), moving 20cm backward in the vertical direction (-20cm, 0, 0); etc.

[0066] The process of training the control model in this embodiment is to achieve autonomous learning of the robotic arm, freeing it from manual control. Therefore, feedback is needed for each movement of the robotic arm to guide it to make the correct action; this feedback is defined as reward information. Positive reward information indicates that the robotic arm's current action is helpful in achieving the goal, while negative information hinders it. In this way, the robotic arm can quickly learn the optimal strategy for achieving the goal through each reward message. The reward information is determined based on the goals of the current environment; when the robotic arm takes an action to achieve one of the goals, the reward value is 1, and otherwise 0.

[0067] Taking dining in the canteen for disabled employees as an example, reward information The reward is set to +1 when the robotic arm reaches the food pick-up window, picks up the food, or safely delivers it to the dining area; otherwise, the reward is 0.

[0068] The first reward value for the interactive action is determined based on the relative position information of the simulated robotic arm and the target location before and after the interactive action, and the relative position information of the simulated robotic arm and the obstacle before and after the interaction. For example, this can be determined based on the change in distance between the simulated robotic arm and the target location before and after the interactive action, and whether the arm is blocked by an obstacle after the interaction. If the simulated robotic arm is closer to the target location after the interactive action and is not blocked by an obstacle, the first reward value is positive; conversely, if it is farther from the target location, the first reward value is negative; and / or if it is blocked by an obstacle, the first reward value is also negative. When the simulated robotic arm reaches the target location in the simulated training environment, it is equivalent to achieving a goal, i.e., the reward value is 1.

[0069] A simulation training environment is constructed based on the target environment in which the target robotic arm is located, and a matching simulation robotic arm is constructed based on the attributes of the simulation training arm, thereby reducing the cost of trial and error learning of the robotic arm in the real environment.

[0070] When setting up the simulation training environment, the simulated robotic arm is first established based on the degrees of freedom and motion dimensions of the target robotic arm. Then, the simulation training environment is built with the robotic arm as the origin and combined with state information, presenting the placement of target objects and obstacles in the actual application scenario, as well as the relative positions of the target objects, obstacles, and target locations to the simulated robotic arm.

[0071] In step S101, the experience pool module is used to store samples generated by the interaction between the simulated robotic arm and the simulation training environment. The sample generated by the i-th interaction between the simulated robotic arm and the simulation training environment is (s i a i r i1 s i+1 ), indicating that the state of the robotic arm before this interaction was s. i When taking interactive action a i Afterwards, the environment will provide the robotic arm with a reward value r. i1 At this point, the robotic arm reaches the next state s. i+1 The samples are used to record one interaction between the robotic arm and the environment. In order to ensure the diversity of samples during training, random sampling is used for each training session, that is, a batch of samples is randomly selected from the experience pool to train the model.

[0072] In this embodiment, 64 samples are randomly sampled from these samples each time to generate a training batch for training the simulated robotic arm. The simulated robotic arm can determine the quality of its current action based on the reward value of these samples, i.e., its contribution to achieving the final goal of safely delivering the food to the dining area.

[0073] In step S102, the action evaluation network module employs a feedforward neural network. Based on the currently selected action of the simulated robotic arm, and combining the current state and the next state, the action evaluation network module uses a feedforward neural network to evaluate the quality of the action; for the samples obtained from the experience pool by the reviewer network, the input of the action evaluation network module is (s i a i s i+1 Then, a second reward value is output to evaluate the quality of the sample action. The second reward value is represented by r. i2 This means that the second reward value will be fed back to the robotic arm as an additional reward along with the first reward value.

[0074] In step S103, the action generation module employs a deep deterministic policy gradient algorithm, combined with an action evaluation network module, to train the optimal model for robotic arm control. This algorithm includes a critic network Q and an actor network μ, where the critic network Q corresponds to a target network Q', and the actor network μ corresponds to a target network μ'. The weights of the critic network Q, actor network μ, target network Q', and target network μ' are denoted by θ. Q θ μ θ Q’ and θ μ‘ .

[0075] The training objective of the motion generation module is the cumulative reward value during the interaction between the maximum simulated robotic arm and the simulated training environment. This cumulative reward value is calculated based on the first reward value and the second reward value. Specifically, the optimization objective of the motion generation module is:

[0076] y i =r i +γQ′(s i+1 ,μ′(s j+1 |θ μ′ )|θ Q′ )

[0077] Wherein, the y i The cumulative reward value after the simulated robotic arm performs i actions is γ, where γ is a discount factor ranging from [0, 1], used to balance the immediate and delayed rewards obtained during the interaction of the simulated robotic arm; i Let r be the reward information for the i-th action. This value is composed of environmental reward information and additional reward information from the action evaluation network module. i =r i1 +r i2 After each action selection made by the simulated robotic arm, the simulation training environment provides feedback r based on that action. i1 That is, the first reward value r i1The action evaluation network module also provides an evaluation of the action based on the action and the states before and after the action selection, namely the second reward value r. i2 The two factors are combined and used as reward information fed back to the robotic arm. In this way, even after the robotic arm makes a series of incorrect motion choices and positive rewards are lacking in the environment, there is still a motion evaluation value to guide its learning.

[0078] Please refer to Figure 2 The Figure 2 This document illustrates a flowchart of a method for controlling a simulated robotic arm to interact with the training environment using the target action, generating a new sample, and storing it in an experience pool module, as described in an embodiment of this application. Specifically, in step S103 of this embodiment, controlling the simulated robotic arm to interact with the training environment using the target action, generating a new sample, and storing it in the experience pool module includes:

[0079] S201. Control the simulated robotic arm to interact with the training environment using the target action, and obtain the state information of the simulated robotic arm before and after the interaction. Based on the state information of the simulated robotic arm before and after the interaction and the pre-configured reward rule information, determine the first reward value fed back by the simulated training environment based on the target action.

[0080] S202. Based on the state information of the simulated robotic arm before interaction, the target action, the state information of the simulated robotic arm after interaction, and the first reward value, generate a new sample and store it in the experience pool module.

[0081] In step S104, the training termination condition includes reaching the required number of training sessions or achieving the training objective.

[0082] Please refer to Figure 3 and Figure 4 The Figure 3 A flowchart illustrating a method for updating the parameters of the commentator network based on the optimization target value is shown. Figure 4 A flowchart illustrating a method for updating the parameters of an actor network based on the action value is provided. Specifically, in step S104, updating the parameters of the commentator network based on the optimization target value includes the following steps S301-S303:

[0083] S301. Calculate the minimum loss function in the commenter network based on the optimization target value, and calculate the gradient of the minimum loss function to determine the first target gradient value;

[0084] S302. Update the parameters of the commentator network based on the first target gradient value.

[0085] In step S104, updating the parameters of the actor network based on the action value includes the following steps S401-S402:

[0086] S401. Calculate the gradient of the action value output by the actor network to determine the gradient value of the second objective.

[0087] S402. Update the parameters of the actor network according to the second target gradient value and the preset exploration bias term, and update the preset exploration bias term according to the second target gradient value.

[0088] In this embodiment of the application, the parameters of the commentator network are the weights of the commentator network; the parameters of the actor network are the weights of the actor network.

[0089] In this embodiment of the application, the minimization loss function is defined as:

[0090]

[0091] The y i Let be the cumulative reward value of the simulated robotic arm after performing i actions; N represents the number of samples collected, used to calculate the loss function; Q(s) i a i |θ Q This represents the evaluation value output by the commenter network after the i-th action is input.

[0092] The gradient of the loss function is calculated to determine the first objective gradient value, which is:

[0093] The parameters of the commentator network are updated based on the first target gradient value, specifically by updating the weights of the commentator network using the following formula:

[0094] θ Q i+1 The weights of the commentator network after the simulated robotic arm performs i actions, i.e., the updated weights; θ Q i The weights of the commentator network before the simulated robotic arm takes i actions are the weights before the update; α is the update step size, ranging from (0, 1), used to control the gradient update step size; This is the gradient value of the first objective.

[0095] The gradient of the action values ​​output by the actor network is calculated to determine the gradient value of the second objective; the gradient value of the second objective is...

[0096] In this embodiment of the application, an exploration bias term θ is introduced to ensure sufficient exploration during the robotic arm's learning process.off This is combined with the weight θ used to update the actor network μ. μ That is, the parameters of the actor network are updated based on the gradient value of the second objective and the preset exploration bias term, specifically by updating the weights of the actor network using the following formula:

[0097]

[0098] Where, θ μ i+1 The weights of the actor network after the simulated robotic arm performs i actions, i.e., the updated weights; θ μ i The weights of the actor network before the simulated robotic arm takes i actions are the weights before the update; α is the update step size, which ranges from (0, 1) and is used to control the gradient update step size. This represents the gradient value for the second objective.

[0099] Meanwhile, during the learning process of the robotic arm, the bias term θ is explored. off The same update method is also used. Where, θ off ′ represents the updated exploration bias, θ off The exploration bias term before the update; exploration bias term θ off The initial value can be set to any value between [0, 1], such as 0, 0.2, 0.65, 1, etc.

[0100] Overfitting is a common problem during the training of robotic arm control models. Overfitting occurs when a trained model performs well on the training set but poorly on the test set. In other words, the control model excessively fits the training data without considering generalization ability. Based on the embodiments of this application, in simple terms, after the reviewer network obtains samples in batches from the experience pool module, it processes the samples according to preset processing rules to obtain processing results, which are then output to the actor network. The actor network determines the target action from a pre-configured action set based on the processing results. Because the actor network lacks generalization ability, when training based on existing samples in the experience pool, it tends to determine the same target action for the same robotic arm state, achieving optimality based on existing samples rather than exploring multiple possible actions. This results in the robotic arm only learning the control process under one (or a limited number of) environmental states during the learning process, without sufficient exploration. In practical applications, the prediction results for new food delivery paths are poor.

[0101] Therefore, this application adopts a synchronously updated exploration bias term. During training, due to the existence of the exploration bias term, the actor network obtains different action values ​​for the same robotic arm state, thus enabling different target actions to be taken. Through trial and error, the optimal path can be explored as much as possible, avoiding overfitting of the robotic arm to the currently collected samples. This encourages the robotic arm to explore different states in the environment. Based on the action evaluation given by the action evaluation network, a path to achieve fast and accurate food delivery is explored, providing greater convenience for the application of robotic arms in real-world scenarios.

[0102] This application also provides a robotic arm control method in its embodiments; please refer to... Figure 5 , Figure 5 A flowchart of the robotic arm control method according to an embodiment of this application is shown; specifically, the control method includes S501-S503:

[0103] S501. Obtain the starting position information, target position information, and target item information of the target robotic arm in the target scene;

[0104] S502. The trained control model generates a set of target actions for the target robotic arm; the set of target actions includes actions of the target robotic arm grasping or putting down target objects, and actions of moving from the starting position to the target position; wherein, the trained control model is trained based on the structure of the target robotic arm and the target scene;

[0105] S503. Control the target robotic arm to execute each action in the target action set in sequence to transfer the target object position to the target location.

[0106] The control model trained in the control method is the control model trained using the training method described in the embodiments of this application.

[0107] Compared to traditional command-controlled robotic arms, the embodiments of this application achieve autonomous control of the robotic arm through deep reinforcement learning algorithms and motion evaluation networks, while reducing the difficulty of manual design. This intelligent control improves the generalization performance of the robotic arm, allowing it to be applied in multiple similar scenarios and increasing the utilization rate of the same robotic arm.

[0108] In this embodiment of the application, a training device for a robotic arm control model is also provided. Please refer to... Figure 6 , Figure 6 A schematic diagram of the structure of the training device for the robotic arm control model according to an embodiment of this application is shown; specifically, the training device for the control model includes:

[0109] The experience pool module 601 is used to store samples generated by each interaction between the pre-built simulation training arm and the simulation training environment; the samples include the state information of the simulation robotic arm before the interaction, the interaction actions taken when interacting with the training environment, the state information of the simulation robotic arm after the interaction, and the first reward value fed back by the simulation training environment based on the interaction actions.

[0110] The motion evaluation network module 602 is used to calculate the second reward value of the interaction action in the sample based on the state information of the simulated robotic arm before interaction, the interaction action taken when interacting with the training environment, and the state information of the simulated robotic arm after interaction.

[0111] The action generation module 603 is used to randomly obtain at least one sample from the experience pool module through the commentator network, process the at least one sample and the second reward value corresponding to the sample according to the preset processing rules to obtain the processing result, and output the processing result to the actor network. The actor network determines the target action and action value from the pre-configured action set according to the processing result, so as to control the simulated robotic arm to interact with the training environment by adopting the target action, generate a new sample and store it in the experience pool module.

[0112] The training module 604 is used to calculate an optimization target value based on the first reward value and the second reward value, update the parameters of the commenter network based on the optimization target value, update the parameters of the actor network based on the action value, until the control model meets the training termination condition, and obtain a trained control model. The trained control model is used to control the target robotic arm to perform target actions.

[0113] The training device provided in this application embodiment randomly obtains samples from the experience pool module, and based on the interactive actions of the samples, as well as the state information of the simulated robotic arm before and after the interaction, calculates the second reward value of the interactive action in the sample through the action evaluation network module; and continuously corrects the control strategy of the robotic arm by using the first reward value and the second reward value of each sample in each training process, and continuously optimizes the weights of the commentator network and the actor network according to the processing results of the commentator network, thereby obtaining a trained control model. The control model autonomously generates a set of target actions according to the structure of the target robotic arm and the target environment in which the target robotic arm is located, realizing the robotic arm The autonomous control reduces the difficulty of manual design, and the intelligent control of the robotic arm improves its generalization performance, allowing it to be applied in multiple similar scenarios and increasing the utilization rate of the same robotic arm. During training, after each action selection by the robotic arm, the environment provides a first reward value based on that action, and the action evaluation network module also provides a second reward value based on that action and the state before and after the action selection. The two are added together as reward information and fed back to the robotic arm. This ensures that even after the robotic arm makes consecutive incorrect action selections and there is a lack of positive rewards in the environment, there is still an action evaluation value to guide its learning, improving training efficiency and the accuracy of the target action set autonomously generated by the control model.

[0114] In some embodiments, the training device for the robotic arm control model further includes:

[0115] The environment perception module is used to collect attribute information of the target robotic arm and scene information of the target scene where the target robotic arm is located through an image acquisition device before storing the samples generated by each interaction between the pre-built simulation training arm and the simulation training environment in the pre-built experience pool module. Based on the attribute information of the target robotic arm and the scene information, the module determines the set of state information, the set of actions, and the reward rule information to construct the simulation robotic arm and the simulation training environment. The scene information includes the position of the target robotic arm and the target location information, target object information, and / or obstacle information of the target scene that interact with the target robotic arm.

[0116] In some embodiments, the training device for the robotic arm control model further includes:

[0117] The construction module is used to construct a simulation training environment based on the structure of the target robotic arm and the target environment in which the target robotic arm is located, before storing the samples generated by each interaction between the pre-built simulation training arm and the simulation training environment in the pre-built experience pool module; the simulation training environment includes the target location, target object and / or obstacle located at a preset position.

[0118] In some embodiments, the reward rule information determined by the environment perception module is used to determine a first reward value for the interactive action based on the relative position information of the simulated robotic arm and the target location before and after the simulated robotic arm performs the interactive action, and the relative position information of the simulated robotic arm and the obstacle before and after the interaction.

[0119] In some embodiments, in the training device for the robotic arm control model, the action generation module, when controlling the simulated robotic arm to interact with the training environment using the target action to generate a new sample and store it in the experience pool module, is specifically used for:

[0120] The simulated robotic arm is controlled to interact with the training environment using the target action, and the state information of the simulated robotic arm before and after the interaction is obtained. Based on the state information of the simulated robotic arm before and after the interaction and the pre-configured reward rule information, the first reward value fed back by the simulated training environment based on the target action is determined.

[0121] Based on the state information of the simulated robotic arm before interaction, the target action, the state information of the simulated robotic arm after interaction, and the first reward value, a new sample is generated and stored in the experience pool module.

[0122] In some embodiments, in the training device for the robotic arm control model, when the training module updates the parameters of the commentator network according to the optimization target value and updates the parameters of the actor network according to the action value, it is specifically used for:

[0123] Based on the optimization target value, the minimum loss function in the commenter network is calculated, and the gradient of the minimum loss function is calculated to determine the first target gradient value;

[0124] Update the parameters of the commentator network based on the first target gradient value;

[0125] Updating the parameters of the actor network based on the action value includes:

[0126] The gradient of the action values ​​output by the actor network is calculated to determine the gradient value of the second objective.

[0127] The parameters of the actor network are updated based on the second objective gradient value and the preset exploration bias term, and the preset exploration bias term is also updated based on the second objective gradient value.

[0128] Please refer to Figure 7In this embodiment of the application, an electronic device 700 is also provided, including: a processor 702, a memory 701 and a bus. The memory 701 stores machine-readable instructions that can be executed by the processor 702. When the electronic device 700 is running, the processor 702 communicates with the memory 701 through the bus. When the machine-readable instructions are executed by the processor 702, the steps of the training method or the control method of the robotic arm are executed.

[0129] In this embodiment of the application, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, performs the steps of the training method or the control method of the robotic arm.

[0130] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0131] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0132] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0133] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a platform server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0134] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A training method for a robotic arm control model, characterized in that, The training method for the control model includes: The system collects attribute information of the target robotic arm and scene information of the target scene where the target robotic arm is located through an image acquisition device. Based on the attribute information of the target robotic arm and the scene information, it determines a set of state information, a set of actions, and reward rule information to construct a simulated robotic arm and a simulated training environment. The scene information includes the position of the target robotic arm and information about the target location, target object, and / or obstacle in the target scene that interacts with the target robotic arm. The reward rule information is used to determine a first reward value for the interactive action based on the relative position information of the simulated robotic arm and the target location before and after the interactive action, and the relative position information of the simulated robotic arm and the obstacle before and after the interaction. The samples generated from each interaction between the pre-built simulation training arm and the simulation training environment are stored in the pre-built experience pool module. The samples include the state information of the simulation robotic arm before the interaction, the interaction actions taken when interacting with the training environment, the state information of the simulation robotic arm after the interaction, and the first reward value fed back by the simulation training environment based on the interaction actions. Based on the state information of the simulated robotic arm before interaction, the interactive actions taken when interacting with the training environment, and the state information of the simulated robotic arm after interaction in the sample, the second reward value of the interactive action in the sample is calculated by the action evaluation network module; the action evaluation network module adopts a feedforward neural network; the action evaluation network module evaluates the quality of the action based on the currently selected action of the simulated robotic arm and combines the current state and the next state using the feedforward neural network. At least one sample is randomly obtained from the experience pool module through the commentator network, and the at least one sample and the second reward value corresponding to the sample are processed according to the preset processing rules to obtain the processing result. The processing result is output to the actor network. The actor network determines the target action and action value from the pre-configured action set according to the processing result, so as to control the simulated robotic arm to interact with the training environment by adopting the target action, generate a new sample and store it in the experience pool module. An optimization target value is calculated based on the first reward value and the second reward value. The parameters of the commenter network are updated based on the optimization target value, and the parameters of the actor network are updated based on the action value, until the control model meets the training termination condition, thus obtaining a trained control model. The trained control model is used to control the target robotic arm to perform target actions.

2. The training method for the robotic arm control model according to claim 1, characterized in that, Before storing the samples generated from each interaction between the pre-built simulation training arm and the simulation training environment in the pre-built experience pool module, the training method further includes: A simulation training environment is constructed based on the structure of the target robotic arm and the target environment in which the target robotic arm is located; the simulation training environment includes the target location, target object and / or obstacle located at a preset position.

3. The training method for the robotic arm control model according to claim 1, characterized in that, Controlling the simulated robotic arm to interact with the training environment using the target action, generating a new sample and storing it in the experience pool module; including: The simulated robotic arm is controlled to interact with the training environment using the target action, and the state information of the simulated robotic arm before and after the interaction is obtained. Based on the state information of the simulated robotic arm before and after the interaction and the pre-configured reward rule information, the first reward value fed back by the simulated training environment based on the target action is determined. Based on the state information of the simulated robotic arm before interaction, the target action, the state information of the simulated robotic arm after interaction, and the first reward value, a new sample is generated and stored in the experience pool module.

4. The training method for the robotic arm control model according to claim 1, characterized in that, The parameters of the commentator network are updated based on the optimization target value, including: Based on the optimization target value, the minimum loss function in the commenter network is calculated, and the gradient of the minimum loss function is calculated to determine the first target gradient value; Update the parameters of the commentator network based on the first target gradient value; Updating the parameters of the actor network based on the action value includes: The gradient of the action values ​​output by the actor network is calculated to determine the gradient value of the second objective. The parameters of the actor network are updated based on the second objective gradient value and the preset exploration bias term, and the preset exploration bias term is also updated based on the second objective gradient value.

5. A robotic arm control method, characterized in that, The control method includes: Obtain the starting position information, destination position information, and target item information of the target robotic arm in the target scene; Based on the starting location information, the target location information, and the target item information, a set of target actions for the target robotic arm is generated using a trained control model. The set of target actions includes actions by which the target robotic arm grasps or releases the target item, and actions by which it moves from the starting position to the target location. The trained control model is obtained by training based on the structure of the target robotic arm and the target scene, and the trained control model is trained using the training method for the robotic arm control model described in any one of claims 1-4. The target robotic arm is controlled to execute each action in the target action set in sequence to transfer the target object to the target location.

6. A training device for a robotic arm control model, characterized in that, The training device for the control model includes: The experience pool module is used to store samples generated by each interaction between the pre-built simulation training arm and the simulation training environment. The samples include the state information of the simulation robotic arm before the interaction, the interaction actions taken when interacting with the training environment, the state information of the simulation robotic arm after the interaction, and the first reward value fed back by the simulation training environment based on the interaction actions. The action evaluation network module is used to calculate the second reward value of the interactive action in the sample based on the state information of the simulated robotic arm before interaction, the interactive action taken when interacting with the training environment, and the state information of the simulated robotic arm after interaction. The action evaluation network module adopts a feedforward neural network. The action evaluation network module evaluates the quality of the action based on the currently selected action of the simulated robotic arm and combines the current state and the next state using the feedforward neural network. The action generation module is used to randomly obtain at least one sample from the experience pool module through the commentator network, process the at least one sample and the second reward value corresponding to the sample according to the preset processing rules to obtain the processing result, and output the processing result to the actor network. The actor network determines the target action and action value from the pre-configured action set according to the processing result, so as to control the simulated robotic arm to interact with the training environment by adopting the target action, generate a new sample and store it in the experience pool module. The training module is used to calculate the optimization target value based on the first reward value and the second reward value, update the parameters of the commenter network based on the optimization target value, update the parameters of the actor network based on the action value, until the control model meets the training termination condition, and obtain the trained control model. The trained control model is used to control the target robotic arm to perform the target action. An environment perception module is used to collect attribute information of the target robotic arm and scene information of the target scene where the target robotic arm is located through an image acquisition device before storing the samples generated by each interaction between the pre-built simulation training arm and the simulation training environment in the pre-built experience pool module. Based on the attribute information of the target robotic arm and the scene information, the module determines a set of state information, a set of actions, and reward rule information to construct the simulation robotic arm and the simulation training environment. The scene information includes the position of the target robotic arm and the target location information, target object information, and / or obstacle information in the target scene that interact with the target robotic arm. The reward rule information determined by the environmental perception module is used to determine the first reward value of the interactive action based on the relative position information of the simulated robotic arm and the target location before and after the interactive action, and the relative position information of the simulated robotic arm and the obstacle before and after the interaction.

7. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the training method for the robotic arm as described in any one of claims 1 to 4 or the control method for the robotic arm as described in claim 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the training method for the robotic arm as described in any one of claims 1 to 4 or the control method as described in claim 5.

Images