Method for training robotic arm control model, robotic arm control method, device and medium

Abstract

An embodiment of the present invention provides a method for training a robotic arm control model, a robotic arm control method, a device, and a medium. The method includes: obtaining a current atomic task sample of the robotic arm, a current action space parameter sample, and a current visual environment parameter sample of the working area of the robotic arm; according to the current atomic task sample, the current action space parameter sample, the current visual environment parameter sample, and the robotic arm control model, obtaining a predicted action space parameter sample of the robotic arm, controlling the robotic arm to run according to the predicted action space parameter sample, and obtaining an updated visual environment parameter sample of the working area of the robotic arm after running; obtaining an execution result according to the updated visual environment information and a preset execution result judgment model, and training the robotic arm control model according to the execution result to obtain a trained robotic arm control model. The present invention improves the training efficiency and accuracy of the robotic arm control model.

Description

Technical Field

[0001] The present invention belongs to the technical field of robotic arm control, and particularly relates to a method for training a robotic arm control model, a robotic arm control method, a device, and a medium. Background Art

[0002] Robotic arms have always been difficult to work in open and complex scenarios like humans. With the development of reinforcement learning technology in artificial intelligence, it has become possible for robotic arms to think and act like humans.

[0003] In related technologies, based on supervised reinforcement learning methods, a large amount of manual operation data can be obtained through devices such as VR, a training set can be constructed according to the manual operation data, and the robotic arm can learn manual actions based on the manual operation data. However, this method has a high data acquisition cost and low efficiency, so it is difficult to be extended to open scenarios.

[0004] In related technologies, common unsupervised reinforcement learning methods usually perform optimal policy learning through random exploration or exploration with a fixed policy. This method requires a large amount of training time and a large amount of data processing, resulting in a high cost for learning the robotic arm control strategy and difficulty in convergence of the robotic arm control model in complex scenarios. Summary of the Invention

[0005] The present invention provides an unsupervised robotic arm control model training method, a robotic arm control method, a device, and a medium based on task semantic decomposition and reinforcement learning technology to solve the problems of high training cost and poor convergence of the robotic arm control model in related technologies.

[0006] In a first aspect, the present invention provides a method for training a robotic arm control model, including:

[0007] Obtaining the current atomic task sample of the robotic arm, the current action space parameter sample, and the current visual environment parameter sample of the robotic arm operation area; the atomic task sample is obtained by decomposing the target task sample to be executed by the robotic arm;

[0008] According to the current atomic task sample, the current action space parameter sample, the current visual environment parameter sample, and the robotic arm control model, obtaining the predicted action space parameter sample of the robotic arm;

[0009] Controlling the robotic arm to run according to the predicted action space parameter sample, and obtaining the updated visual environment parameter sample of the robotic arm operation area after running;

[0010] Obtaining an execution result according to the updated visual environment information and a preset execution result judgment model, and training the robotic arm control model according to the execution result to obtain a trained robotic arm control model.

[0011] Second aspect, the present invention provides a manipulator control method, including:

[0012] Obtain the current atomic task of the manipulator, the current action space parameters, and the current visual environment parameters of the manipulator working area; the atomic task is obtained by decomposing the target task to be executed by the manipulator;

[0013] According to the atomic task, the current action space parameters, the current visual environment parameters, and the trained manipulator control model, obtain the predicted action space parameters of the manipulator;

[0014] Control the operation of the manipulator according to the predicted action space parameters; the manipulator control model is obtained according to the manipulator control model training method described in the first aspect.

[0015] Third aspect, the present invention provides a manipulator control model training device, the device includes:

[0016] The first acquisition module is used to acquire the current atomic task sample of the manipulator, the current action space parameter sample, and the current visual environment parameter sample of the manipulator working area; the atomic task sample is obtained by decomposing the target task sample to be executed by the manipulator;

[0017] The second acquisition module is used to obtain the predicted action space parameter sample of the manipulator according to the current atomic task sample, the current action space parameter sample, the current visual environment parameter sample, and the manipulator control model;

[0018] The third acquisition module is used to control the operation of the manipulator according to the predicted action space parameter sample and obtain the updated visual environment parameter sample of the manipulator working area after operation;

[0019] The fourth acquisition module is used to obtain the execution result according to the updated visual environment information and the preset execution result judgment model, and train the manipulator control model according to the execution result to obtain the trained manipulator control model.

[0020] Fourth aspect, the present invention provides a manipulator control device, the device includes:

[0021] The fifth acquisition module is used to acquire the current atomic task of the manipulator, the current action space parameters, and the current visual environment parameters of the manipulator working area; the atomic task is obtained by decomposing the target task to be executed by the manipulator;

[0022] The sixth acquisition module is used to obtain the predicted action space parameters of the manipulator according to the atomic task, the current action space parameters, the current visual environment parameters, and the trained manipulator control model;

[0023] A seventh acquisition module, configured to control the operation of the robotic arm according to the predicted action space parameters; the robotic arm control model is obtained by the robotic arm control model training method according to any one of the first aspect.

[0024] In a fifth aspect, the present invention provides an electronic device, including: a processor, a memory, and a computer program stored on the memory and executable on the processor, where when the processor executes the program, it implements the robotic arm control model training method according to the first aspect, or the robotic arm control method according to the second aspect.

[0025] In a sixth aspect, the present invention provides a readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, enabling the electronic device to execute the robotic arm control model training method according to the first aspect, or the robotic arm control method according to the second aspect.

[0026] In summary, by predicting the predicted action space parameter samples of the robotic arm and training the model according to the predicted action space parameter samples, compared with the method of joint angle prediction in related technologies, the non-linear joint angle prediction is avoided, which is convenient for the model to converge. By decomposing the target task samples to obtain atomic task samples and training the robotic arm control model for each atomic task sample respectively, the obtained robotic arm control model has high accuracy. This embodiment does not require obtaining a large amount of manual operation data and using the manual operation data as samples to train the model. Therefore, the acquisition step of training samples is saved, the data processing amount is reduced, and the model training efficiency is improved. Description of the Drawings

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0028] Figure 1 is a flowchart of the steps of a robotic arm control model training method provided by an embodiment of the present invention;

[0029] Figure 2 is another schematic diagram of a robotic arm provided by an embodiment of the present invention;

[0030] Figure 3 is a flowchart of the steps of a robotic arm control model training method provided by an embodiment of the present invention;

[0031] Figure 4 is a flowchart of the steps of a robotic arm control method provided by an embodiment of the present invention;

[0032] Figure 5 is a flowchart of steps of a method for training a robotic arm control model provided by an embodiment of the present invention;

[0033] Figure 6 is a flowchart of steps of a method for training a robotic arm control model provided by an embodiment of the present invention;

[0034] Figure 7 is a structural diagram of a device for training a robotic arm control model provided by an embodiment of the present invention;

[0035] Figure 8 is a structural diagram of a robotic arm control device provided by an embodiment of the present invention;

[0036] Figure 9 is a structural diagram of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0037] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention. '

[0038] Next, in conjunction with the relevant drawings and embodiments, the method for training a robotic arm control model of the present application will be further described: <\

[0039] Figure 1 is a flowchart of steps of a method for training a robotic arm control model provided by an embodiment of the present invention. As Figure 1 shown, the method may include:

[0040] Step 101, obtaining the current atomic task sample of the robotic arm, the current action space parameter sample, and the current visual environment parameter sample of the working area of the robotic arm.

[0041] Among them, the atomic task sample is obtained by decomposing the target task sample to be executed by the robotic arm. '

[0042] Specifically, the atomic task sample is the semantic encoding of the task obtained by decomposing the target task sample, and the target task sample is used to describe the task sample to be executed by the robotic arm.

[0043] For example, if the task to be executed by the robotic arm is to move an object from position A to position B, the task to be executed can be decomposed to obtain the following atomic tasks: the robotic arm moves from the initial position to position A and grasps the object; the robotic arm picks up the object and places it at position B.

[0044] Among them, the visual environment parameter samples include the depth information and visual features of the operation area. For example, through a 3D reconstruction algorithm, 3D environment information of the robotic arm operation area can be constructed. The 3D environment information includes depth information, and through the trained visual recognition algorithm and depth information, visual features can be obtained.

[0045] The action space parameter samples are a set of robotic arm action parameters in a given operation environment. The action space parameter samples reflect the action parameters of the robotic arm. For example, they can include pose parameters and the predicted value of the rotation angle of the robotic arm's upper arm around the shoulder-wrist connection line.

[0046] Step 102: According to the current atomic task sample, the current action space parameter sample, the current visual environment parameter sample, and the robotic arm control model, obtain the predicted action space parameter sample of the robotic arm.

[0047] For example, the robotic arm control model can include an action prediction model. Among them, the action space parameter calculation model can be a Deep Deterministic Policy Gradient (DDPG) model or an Asynchronous Advantage Actor-Critic (A3C) model. For example, input the current atomic task sample, the current action space parameter sample, and the current visual environment parameter sample into the action prediction model to obtain the predicted action space parameter sample of the robotic arm.

[0048] Specifically, input the atomic task sample, the current action space parameter sample, and the current visual environment parameter sample into the robotic arm control model to obtain the predicted action space parameters predicted by the robotic arm control model.

[0049] Step 103: Control the operation of the robotic arm according to the predicted action space parameter sample, and obtain the updated visual environment parameter sample of the robotic arm operation area after operation.

[0050] Furthermore, obtain the action sequence of joint rotation parameters according to the predicted action space parameter sample, and control the operation of the robotic arm based on the action sequence of joint rotation parameters.

[0051] For example, calculate the rotation parameters of each joint of the robotic arm according to the predicted action space parameter, construct the action sequence of the robotic arm joint rotation parameters according to the rotation parameters of each joint, and control the operation of the robotic arm according to the action sequence of the joint rotation parameters. Among them, referring to Figure 2 , the joint rotation parameter is the rotation angle of the robotic arm joint. In Figure 2 , r represents the rotation angle of the robotic arm joint, ∑0 is the horizontal plane coordinate system, and ∑7 is the reference coordinate system of the 7th robotic arm joint.

[0052] The robotic arm operates according to the predicted motion space parameters. After the operation, the visual environment captured by the sensors on the robotic arm will also change. Correspondingly, the visual environment parameter samples in the operation area of the robotic arm after operation will also change, changing from the current visual environment parameter samples in step 101 to updated visual environment parameter samples.

[0053] Step 104: Obtain the execution result according to the updated visual environment information and the preset execution result judgment model, and train the robotic arm control model according to the execution result to obtain the trained robotic arm control model.

[0054] The execution result includes task completion or non-completion. For example, if the atomic task described by the atomic task sample is for the robotic arm to grasp an object at position A, then after the robotic arm operates according to the predicted motion space parameter sample, if the robotic arm grasps the object from position A, it indicates that the execution result is task completion; otherwise, the execution result is task non-completion.

[0055] Exemplarily, the updated visual environment information may include the depth information and visual features in the updated visual environment after the robotic arm operates according to the current motion space parameter sample.

[0056] Exemplarily, the preset execution result judgment model may be a convolutional neural network model (YouOnly LookOnce, YOLO).

[0057] Exemplarily, according to the correspondence between the preset execution result and the preset reward value, obtain the reward value corresponding to the execution result, and train the robotic arm control model according to the reward value to obtain the trained robotic arm control model.

[0058] With the development of large model technology, various applications based on large models emerge in an endless stream. Embodied intelligence is a major application direction of large models. In the field of embodied intelligence, how to perform robotic arm operations in open and complex scenarios has always been a hot topic and a difficult problem in this field.

[0059] In this embodiment, obtaining the predicted motion space parameter samples of the predicted robotic arm and training the model according to the predicted motion space parameter samples, compared with the method of training the model by predicting joint angles in related technologies, this embodiment avoids non-linear joint angle prediction and facilitates model convergence. By decomposing the target task samples into atomic task samples and training the robotic arm control model for each atomic task sample respectively, the obtained robotic arm control model has high accuracy. This embodiment does not require obtaining a large amount of manual operation data and using the manual operation data as samples to train the model. Therefore, it saves the acquisition steps of training samples, reduces the data processing volume, and improves the model training efficiency.

[0060] In addition, this embodiment is applicable to the operation of a robotic arm in an open scenario. By decomposing the complex grasping tasks of the robotic arm in the open scenario into atomic tasks and training a model for the atomic tasks, it is possible to avoid separately designing a reward function and a state space for each grasping task of the robotic arm, avoid repetitive operation steps in model design, and improve the generalization and expandability of the robotic arm control model.

[0061] Referring to Figure 3 , the method for training a robotic arm control model may include the following steps:

[0062] Step 201: Obtain a current visual environment parameter sample of the robotic arm operation area.

[0063] Among them, the atomic task sample is obtained by decomposing the target task sample to be executed by the robotic arm.

[0064] The method in this step has been described in the foregoing step 101 and will not be elaborated here.

[0065] Step 202: Obtain a current joint parameter sample of the robotic arm joints.

[0066] Among them, the current joint parameter sample may include the rotation angles of each robotic arm joint relative to the initial state when the robotic arm is in the current state.

[0067] Step 203: Input the current joint parameter sample into a preset action space parameter calculation model to obtain a current action space parameter sample.

[0068] Among them, the current action space parameter sample includes the pose information of the end of the robotic arm and the rotation angle of the robotic arm's upper arm around the shoulder-wrist connection line.

[0069] In this step, the current action space parameter sample includes the rotation angle of the robotic arm's upper arm around the shoulder-wrist connection line. On the basis of the end coordinate information and angle information of the robotic arm, adding the rotation angle of the upper arm around the shoulder-wrist joint connection line constitutes the action space parameter sample in this embodiment. Subsequently, through this current action space parameter sample and the preset activity range of the target joint, it is possible to achieve a unique mapping between the action space and the robotic arm joint angles on the premise that the action space is reachable, avoiding the situation of infinitely many solutions for redundant robotic arms when mapping the end pose of a multi-axis (such as a 7-axis) robotic arm to joint parameters. At the same time, it is possible to simplify the problem of non-linear joint angle control to reach the target pose, facilitating the convergence of the robotic arm control model.

[0070] Step 204: Input the atomic task sample into a preset semantic model to obtain a semantic encoding sample of the atomic task sample.

[0071] Among them, the semantic encoding sample is used to reflect the semantic content of the atomic task sample.

[0072] Among them, the semantic model can be a Transformer-based bidirectional encoder model (Bidirectional Encoder Representations from Transformers, BERT), or a large language model (Large Language Model, LLM), or other semantic models.

[0073] Step 205: Obtain a predicted action space parameter sample of the robotic arm according to the semantic encoding sample, the current action space parameter sample, the current visual environment parameter, and the robotic arm control model.

[0074] The predicted action space parameter sample includes at least one of the following: the predicted value of the pose at the end of the robotic arm, and the predicted value of the rotation angle of the robotic arm's upper arm around the shoulder-wrist connection line.

[0075] Specifically, input the semantic encoding sample, the current action space parameter sample, and the current visual environment parameter into the robotic arm control model to obtain the predicted action space parameter sample of the robotic arm.

[0076] Step 206: Obtain the preset activity range of the target joint among the robotic arm joints.

[0077] Exemplarily, referring to Figure 2 , the robotic arm includes 7 robotic arm joints. Among them, the elbow joint is determined as the target joint, and the preset activity range of the target joint is set to 0 to 180 degrees.

[0078] Step 207: Using the preset activity range as a constraint condition, according to the predicted action space parameter sample and the preset joint parameter calculation model, obtain the joint parameter sample of the robotic arm joints.

[0079] Exemplarily, the preset joint parameter calculation model can be the forward and inverse kinematic equations. Taking the predicted action space parameter sample as the input parameter of the forward and inverse kinematic equations, and using the preset activity range as a constraint condition, obtain the joint parameters calculated by the forward and inverse kinematic equations.

[0080] The predicted action space parameter sample includes the pose parameters at the end of the robotic arm. The robotic arm includes multiple robotic arm joints. Substituting the predicted action space sample into the forward and inverse kinematic equations can obtain the joint parameters of each robotic arm joint.

[0081] Step 208: Control the operation of the robotic arm according to the joint parameter sample.

[0082] Exemplarily, the joint parameter sample includes the rotation angles of each robotic arm joint. According to the rotation angles of each robotic arm joint, the robotic arm is controlled to operate so that the rotation angles of each robotic arm joint are equal to the rotation angles of each robotic arm joint in the joint parameter sample.

[0083] Taking the preset activity range of the target joint as a constraint, based on the predicted action space parameter sample and the preset joint parameter calculation model, the joint parameter sample of the robotic arm joint is obtained. Thus, it can be ensured that the joint parameters included in the obtained joint parameter sample satisfy the constraint of the preset activity range of the target joint, and it can be ensured that the joint parameter sample meets the operation requirements of the robotic arm.

[0084] Step 209, obtain the updated visual environment parameter sample of the robotic arm working area after controlling the robotic arm to operate according to the joint parameter sample.

[0085] Among them, the updated environmental information includes the depth information and visual features of the robotic arm working area after the robotic arm operates according to the joint parameter sample in step 208.

[0086] Exemplarily, after controlling the robotic arm to operate, the information collected by the sensors in the robotic arm for collecting environmental information also changes. Correspondingly, the visual environment information of the robotic arm working area also changes. Further, the method of step 101 above can be referred to obtain the updated depth information and visual features of the robotic arm working area after the robotic arm operates.

[0087] Step 210, obtain the execution result according to the updated visual environment information and the preset execution result judgment model, and train the robotic arm control model according to the execution result to obtain the trained robotic arm control model.

[0088] Among them, the execution result includes: the atomic task is completed or not completed.

[0089] Exemplarily, the preset action recognition model can be a YOLO model. Samples of environmental information with known execution results can be obtained, and the execution result is used as a label to train the YOLO model with the samples and labels to obtain the preset action recognition model.

[0090] According to the updated visual environment parameter sample and the preset action recognition model, the execution result of the atomic task can be quickly obtained.

[0091] Exemplarily, the atomic task includes a plurality of decomposition actions to be executed in sequence. Step 205 may include the following sub-steps:

[0092] Sub-step 2051, according to the current decomposition action, the current action space parameter sample, the current visual environment parameter sample, and the robotic arm control model, obtain the current predicted action space parameter sample of the robotic arm.

[0093] For example, the atomic task is that the robotic arm grasps an object at position A from the initial position, and this atomic task can be decomposed into multiple decomposition actions.

[0094] According to the current decomposition action, the current action space parameter sample, the current visual environment parameter sample, and the robotic arm control model, obtain the current predicted action space parameter sample of the robotic arm. Then, the obtained predicted action space parameter sample is the predicted action space parameter sample for the current decomposition action. Subsequently, after the atomic task to which the decomposition action belongs is completed, the robotic arm control model can be trained based on the predicted action space parameter sample of the decomposition action.

[0095] For example, this atomic task can be decomposed into 20 decomposition actions, and each decomposition action corresponds to a segment of the path that the robotic arm moves.

[0096] Correspondingly, step 210 may include the following sub-steps:

[0097] Sub-step 2101, obtain the number of executed decomposition actions that have been executed.

[0098] For example, for each executed decomposition action, perform an increment operation on the number of executed decomposition actions to obtain the number of executed actions.

[0099] Sub-step 2102, when the number of executed actions is less than or equal to the total number of all decomposition actions, and it is determined that the robotic arm has completed the current atomic task according to the updated visual environment information and the preset execution result judgment model, determine that the execution result is task completed.

[0100] When the number of executed actions is less than or equal to the total number of all decomposition actions, and it is determined that the robotic arm has completed the current atomic task according to the current predicted action space parameter, it means that not all decomposition actions have been executed, the robotic arm has completed the current atomic task, or just all decomposition actions have been executed and the robotic arm has completed the current atomic task.

[0101] In this case, it means that the robotic arm has completed the current atomic task without exceeding the set number of executions (i.e., the total number of decomposition actions), and the operation of the robotic arm meets the expectations. Therefore, in this case, determine that the execution result is task completed.

[0102] Sub-step 2103, when the number of executed actions is greater than the total number of all decomposition actions, or the robotic arm cannot reach the position corresponding to the current predicted action space parameter sample, determine that the execution result is task not completed.

[0103] When the number of executions is greater than the total number of all decomposed actions, the number of executions representing the setting of the number of times the robotic arm executes an action (i.e., the total number of decomposed actions) is used. In this case, the operation of the task robotic arm does not meet the expectations, and the execution result is determined to be that the task is not completed.

[0104] The total number of decomposed actions is equivalent to the upper limit value of the number of actions to be executed to complete the atomic task, and the number of executed actions is equivalent to the number of actions that have been executed. Based on the number of executed decomposed actions, it is judged whether the execution result of the current atomic task is that the task is completed or not completed, and this judgment result can meet the requirements for the number of operations of the robotic arm.

[0105] In the case where the robotic arm cannot reach the position corresponding to the current predicted action space parameter, it means that the robotic arm cannot complete the action corresponding to the predicted action space parameter. Therefore, the execution result is determined to be that the task is not completed.

[0106] Correspondingly, step 210 may include the following sub-steps:

[0107] Sub-step 2104, when the execution result is that the task is completed, obtain the predicted action space parameter samples, environmental information, and reward values respectively corresponding to each executed decomposed action.

[0108] Exemplarily, for each decomposed action, the execution result is obtained according to the methods of steps 202 to 210, as well as the predicted action space parameter samples and environmental information during the process of obtaining the execution result. Among them, according to the corresponding relationship between the preset execution result and the preset reward value, the reward value of each decomposed action is obtained.

[0109] Among them, obtaining the reward value of the decomposed action may include the following sub-steps: <X

[0110] Sub-step A1, obtain the execution order of the decomposed action among the multiple decomposed actions of the atomic task.

[0111] Exemplarily, the atomic task is decomposed to obtain multiple decomposed actions that need to be executed in sequence. Among them, each decomposed action has a corresponding execution order.

[0112] Sub-step A2, determine the attenuation coefficient of the decomposed action according to the execution order.

[0113] Exemplarily, the attenuation coefficient is positively correlated with the execution order, that is, the larger the execution order, the larger the attenuation coefficient, and vice versa, the smaller the execution order, the smaller the attenuation coefficient.

[0114] Sub-step A3, obtain the product of the attenuation coefficient and the preset reward value, and determine the product as the reward value of the decomposed action.

[0115] Among them, the preset reward value is related to the execution result. Further, when the execution result is that the task is completed, the preset reward value is Q, and Q > 0. When the execution result is that the task is not completed, the preset reward value is -Q.

[0116] Among them, the preset reward value can be set according to user needs. For example, it can be set to 10000, 1000, or other values.

[0117] Exemplarily, the product of the decay coefficient related to the execution order of the decomposition actions and the preset reward value is determined as the reward function. The decay coefficient is determined according to the execution order of the decomposition actions, and the reward value of the decomposition action is determined according to the product of the decay coefficient and the preset reward value. This reward value can reflect the influence degree of the decomposition action on the prediction result.

[0118] In one embodiment, the preset reward value is related to the execution result. The execution result is the result obtained based on visual recognition. The reward function is obtained based on the preset reward value, which realizes the use of the reward function based on visual recognition and improves the scalability of the reward function in the open scenario.

[0119] Sub-step 2105: Train the robotic arm control model according to the predicted action space parameter samples and reward values corresponding to each decomposition action, and obtain the trained robotic arm control model.

[0120] Obtain the predicted action space parameter samples, environmental information, and reward values of each decomposition action. Train the robotic arm control model according to the predicted action space parameter samples and reward values corresponding to each decomposition action, and obtain the robotic arm control model, which is a model related to each decomposition action. When using this robotic arm control model for robotic arm control subsequently, the action space parameters of each decomposition action can be obtained, and the robotic arm can be controlled to operate based on the action space parameters of each decomposition action. This control method has the advantage of high control accuracy.

[0121] Correspondingly, step 210 may include the following sub-steps:

[0122] Sub-step 2106: Train the robotic arm control model according to the execution result of the atomic task, and obtain the trained robotic arm control model for the atomic task.

[0123] Exemplarily, for each atomic task, the method of steps 201 to 209 is used to obtain the trained robotic arm control model for the atomic task.

[0124] Sub-step 2107: Construct a robotic arm control model for executing the target task according to the trained robotic arm control models of multiple atomic tasks.

[0125] Exemplarily, the overall robotic arm control model of multiple atomic tasks is determined as the robotic arm control model for performing the target task.

[0126] Exemplarily, it is also possible to obtain the execution results of decomposed actions, environmental information, and predicted action space parameters in all atomic tasks, and train the robotic arm control model based on this data to obtain the trained robotic arm control model.

[0127] In the related art, the advanced robotic arm control method is the imitation learning method. However, the generation of exemplary data is the bottleneck of this method. In the method of the related art, it is necessary to obtain a large amount of manual operation data, let the robotic arm learn the manual operation actions, and obtain the robotic arm control model; it is also possible to construct a massive robot action training set through manual operations. However, since the manual construction of training data is slow, it will affect the model training efficiency. In addition, in the related art, the robotic arm joint angles and the relative position between the target object and the robotic arm are used as the observation space, the robotic arm joint angles are used as the action space, and the reward function is calculated based on the distance between the target object and the robotic arm. In the embodiments of the present application, the environmental information including depth information, the pose parameters of the robotic arm, and the atomic task semantic encoding can be used as the observation space. The robotic arm pose parameters are used as the action space, and the reward value corresponding to the execution result judged by the artificial intelligence model is used as the reward function. Based on this information, the robotic arm control model can be trained quickly.

[0128] In the related art, the reward function for training the robotic arm control model may include the main line reward, the approaching target reward, the approaching obstacle penalty, and the obstacle movement penalty. The state space is the pose distance between the target object and the robotic arm and the minimum circumscribed sphere of all obstacles, and the action space is the angles of each joint and the end pose calculated from the angles. The model of this method predicts the joint angles. In this embodiment, the action space does not include the angles of each joint. Therefore, the end pose of the action space is not calculated from the angles of each joint, but directly predicted by the intelligent model. Further, the state space of this embodiment is the depth information of the entire environment, the context semantic information based on the up and down, and the pose information, and the reward function of this embodiment is the grasping state of visual recognition. Based on the predicted action space parameter samples, the problem that the model is difficult to converge caused by the non-linear joint angles can be avoided.

[0129] In related technologies, a model can also be constructed through a learning method of imitating real examples. Among them, all training data are pre-constructed standard data. The large model used is a vision large model, and the model features include those of a vision segmentation model, a vision-text alignment model, etc. In this embodiment, through a self-supervised reinforcement learning method without action annotation, a semantic or multi-modal large model can be used, and features such as depth information features, vision recognition features, and atomic task text encoding are used for model training, and the model training efficiency is higher.

[0130] Referring to Figure 4 , an embodiment of the present application further provides a robotic arm control method, and the method includes:

[0131] Step 301, obtain the current atomic task of the robotic arm, the current action space parameters, and the current visual environment parameters of the robotic arm working area.

[0132] Among them, the atomic task is obtained by decomposing the target task to be executed by the robotic arm. The method of this step can refer to the description of the foregoing step 101 and will not be elaborated here.

[0133] Step 302, according to the atomic task, the current action space parameters, the current visual environment parameters, and the trained robotic arm control model, obtain the predicted action space parameters of the robotic arm.

[0134] Exemplarily, the atomic task, the current action space parameters, and the current visual environment parameters are input into the trained robotic arm control model to obtain the predicted action space parameters of the robotic arm.

[0135] Among them, the action space parameters may include the coordinate information and angle information of the end of the robotic arm, and the rotation angle of the robotic arm's upper arm around the shoulder-wrist connection line.

[0136] Step 303, control the operation of the robotic arm according to the predicted action space parameters.

[0137] Exemplarily, according to the predicted action parameters, obtain the joint parameters of each joint of the robotic arm, and control the operation of the robotic arm based on the action sequence of the joint rotation parameters formed by the joint parameters of each robotic arm joint. Among them, the robotic arm control model is obtained according to the robotic arm control model training method of any of the foregoing embodiments.

[0138] In this embodiment, by predicting the predicted action space parameter samples of the robotic arm and training the model according to the predicted action space parameter samples, compared with the method of joint angle prediction in the related art, the non-linear joint angle prediction is avoided, which is convenient for the model to converge. By decomposing the target task samples, atomic task samples are obtained, and the robotic arm control model is trained for each atomic task sample respectively, and the obtained robotic arm control model has high accuracy. Based on this robotic arm control model, the pose parameters of the robotic arm are predicted, and based on the pose parameters, the operation of the robotic arm can be accurately controlled.

[0139] Exemplarily, the robotic arm control model training system of the present application includes the following parts: an environment perception module, an action determination module, an instruction semantics module, a pose calculation module, and an action model.

[0140] Among them, the action model is used to output the target pose according to the current state characteristics of the robotic arm, the environment characteristics output by the environment model, and the atomic task encoding of the instruction semantics module, generate the action parameters of the robotic arm through pose calculation, control the operation of the robotic arm according to the action parameters through the action model, and then iteratively update the action model weights through the reward model. Further, the environment perception module is used to obtain visual environment parameter information; the action determination module is equivalent to the execution result judgment model in the foregoing embodiment, and is used to judge the execution result of the robotic arm performing the task according to the visual environment parameter information; the instruction semantics module is equivalent to the preset semantic model in the foregoing embodiment, and is used to convert the atomic task into a semantic encoding; the pose calculation module is equivalent to the action space parameter calculation model in the foregoing embodiment, and is used to predict the action space parameters of the robotic arm; the action model is used to control the operation of the robotic arm.

[0141] In this embodiment, the input parameters of the robotic arm control model are the observation value of the environment and the task instruction. Among them, the observation value of the environment is equivalent to the visual environment parameter sample in the foregoing embodiment, and the task instruction is equivalent to the semantic encoding in the foregoing embodiment.

[0142] In the inference stage of the robotic arm control model, action parameters are continuously output to the robotic arm until the task is completed or fails. In the training stage of the robotic arm control model, after each action of the robotic arm, the reward is calculated through the reward model to update the model.

[0143] Specifically, the environment perception module is used to calculate the depth information and semantic information of the environment of the robotic arm operation area through the trained Artificial Intelligence (AI) model, so as to use the depth information and semantic information as the feature input of the action model in the subsequent processing process.

[0144] Observation values of the environment of the operation area can be obtained through a monocular camera, a multi-camera, a depth camera, a lidar, etc., and input into the environmental perception module to obtain the depth information and semantic information of the environment. Further, 3D environmental information can be constructed through a 3D reconstruction algorithm, and visual feature R can be obtained through a pre-trained visual recognition algorithm. Among them, R is the output of the neural network before the last fully connected layer of the visual model.

[0145] The instruction semantic module is used to obtain the context information of the instruction. The instruction semantic module can be a semantic large model or a multi-modal large model. The instruction semantic module is used to decompose any task into atomic tasks in the atomic task list. Further, a task instruction can be sent to the instruction semantic module from the task list and input into the instruction semantic module to obtain the context information obtained by the instruction semantic module according to the task instruction.

[0146] Exemplarily, through the instruction semantic model, the atomic task that needs to be performed currently and has not been completed can be selected from the atomic task list, and its corresponding semantic encoding E is output.

[0147] The action determination module is used to judge whether the atomic task is completed through a trained or fine-tuned AI model. Specifically, the action determination model is used to obtain the execution result of the atomic task after the current atomic task is executed. According to the execution result, a reward value is obtained. When the reward value is greater than the preset reward value threshold, the execution result of the atomic task is regarded as the successful execution of the task, otherwise the atomic task status is recorded as being executed, and the context information of the instruction semantic module is updated.

[0148] Specifically, the semantic encoding of the atomic task and the picture of the operation area after the robotic arm moves are input into the action determination model to obtain the action execution result. Exemplarily, when the action is completed, the execution result identifier 1 is output, otherwise 0 is output.

[0149] In this embodiment, the robotic arm control model constructed through reinforcement learning indirectly controls the movement of the robotic arm by predicting the pose parameters of the robotic arm at the current moment through the pose calculation module. The pose calculation module is used to obtain the predicted action space parameter information of the end of the robotic arm according to the environmental depth information D, the image feature R, and the atomic task semantic encoding output by the instruction semantic module.

[0150] The predicted action space parameter information includes the end position (x, y, z) of the robotic arm, the end pose (Y, P, R), and the angle r of the robotic arm's upper arm rotating around the shoulder-wrist connection line.

[0151] The observation input of the motion model, the semantic input of the atomic instruction, the output pose parameters, and the reward value corresponding to the execution result after running the joint parameters corresponding to the predicted pose of the robotic arm can be used as training data to iteratively optimize the model parameters of the robotic arm control model until the model converges. Exemplarily, during the model training phase, a random perturbation can be added to each variable output by the model.

[0152] The following refers to Figure 5 and Figure 6 , and further illustrate the robotic arm control model training method of the present application:

[0153] In one embodiment, the robotic arm control method may include the following steps:

[0154] Step S1, initialize the context of the atomic task in the instruction semantics module, initialize the environmental information, obtain the environmental information t at the current moment, and set the task to be executed.

[0155] The environmental information t in this step is the Figure 6 state t in

[0156] Obtain the task instruction of the robotic arm, initialize the context information of the task instruction, initialize the environmental information of the working area, obtain the visual environmental parameters at the current moment, and set the task that the robotic arm is currently to execute. Among them, the instruction semantics module can decompose the task instruction into a set of atomic tasks in the list through the semantic large model, and encode the atomic tasks into semantic features. Then, according to the current atomic task completion status, send the encoded atomic task to be executed to the motion model.

[0157] Step S2, through the instruction semantics module, according to the task instruction and the context status information of the task instruction, select the atomic task that needs to be executed and is not completed from the list of atomic tasks corresponding to the task to be executed currently, and output the semantic encoding E of the atomic task.

[0158] Among them, the context information of the task instruction can reflect the task executed by the previous task instruction of the current task instruction and the task to be executed by the next task instruction.

[0159] Exemplarily, a grasping scene task can be captured and an atomic task list can be designed. Among them, the atomic task list includes multiple atomic tasks that need to be executed in sequence.

[0160] Exemplarily, according to predefined atomic tasks, the recorded actions are segmented to obtain multiple segmented segments. Each segmented segment contains a set of sequences, where each element of each set of sequences includes: the state before action, the current state including the visual observation value of the robotic arm camera, the joint angles before action, the atomic task identifier, the executed action (for example, the executed joint rotation angle), and the state after the executed action. Among them, the state after the executed action can be obtained through the visual observation value of the camera, and the state after the executed action includes whether the atomic task is completed or not.

[0161] Step S3: Through the environmental perception module, according to the sensor for obtaining environmental information t, obtain the depth information D and visual features R corresponding to the environmental information t.

[0162] The environmental information t in this step is equivalent to Figure 6 the environmental features obtained through the environmental perception module in

[0163] Exemplarily, a monocular camera can be installed at the center of the gripper of the robotic arm to obtain environmental information through the monocular camera. An RGB-D (Red Green Blue Depth) camera is installed at a fixed position in the environment without obvious dead angles to obtain environmental information to ensure there are no overly large visual dead angles.

[0164] Step S4: Through the pose calculation module, obtain the current joint parameters T of each joint of the robotic arm, and based on the forward and inverse kinematic equations, obtain the pose parameters W of the robotic arm.

[0165] Among them, W includes the end position (x, y, z), the end pose (Y, P, R), and the angle r of the upper arm rotating around the shoulder-wrist connection line. x, y, and z respectively represent the coordinates on the X-axis, Y-axis, and Z-axis in the reference coordinate system, and Y, P, and R respectively represent the yaw angle, pitch angle, and roll angle.

[0166] Based on the 7 joint parameters of the robotic arm, calculate the end position (x, y, z), the end pose (Y, P, R), and the angle r of the upper arm of the robotic arm rotating around the shoulder-wrist connection line.

[0167] Step S5: Through the action model, obtain the semantic encoding E, the depth information D, the visual features R, and the pose parameters W of the robotic arm, and based on the pose prediction model, obtain the predicted pose parameters W1 of the robotic arm.

[0168] Among them, W1 includes the coordinates (x1, y1, z1) of the end of the robotic arm, the end pose (Y1, P1, R1), and the angle r1 of the upper arm rotating around the shoulder-wrist connection line.

[0169] x1, y1, and z1 respectively represent the coordinates of the X-axis, Y-axis, and Z-axis in the reference coordinate system, and Y1, P1, and R1 respectively represent the yaw angle, pitch angle, and roll angle. In step S6, the pose calculation module obtains the pose prediction parameter W1. With the movement range of the elbow joint opening and closing as the constraint, through the forward and inverse kinematic equations, the joint parameters T1 of each joint of the robotic arm are calculated.

[0170] Specifically, the robotic arm has 7 joints. Among them, the elbow joint is the 4th joint. The pose calculation module obtains the output W1 of the motion model and restricts the preset movement range of the 4th joint (elbow joint opening and closing) among the 7 joints to 0 - 180 degrees. Based on the preset movement range, the predicted motion space parameters, and the forward and inverse kinematic equations, the new 7 joint parameters T1 are calculated.

[0171] The pose calculation module is used to convert the pose parameters and joint parameters of the robotic arm through numerical calculation, so as to convert the motion space of the motion model from the original joint parameters to pose parameters, promoting the convergence of the motion model.

[0172] The process of converting pose to joint parameters is as follows: Restrict the movement range of joint 4 (elbow joint opening and closing) to 0 - 180 degrees. Input the end position (x, y, z), end pose (Y, P, R) of the robotic arm predicted by the motion model, and the angle r of the robotic arm's upper arm rotating around the shoulder-wrist connection line (see Figure 2 ).

[0173] Among them, according to the physical structure of the robotic arm, the values of each joint of the robotic arm are calculated. If there is a solution, the unique solution is output; otherwise, no solution is output.

[0174] In step S7, it is judged whether the forward and inverse kinematic equations have a solution. If there is a solution, go to step S8; otherwise, go to step S12.

[0175] The forward and inverse kinematic equations having a solution means that the robotic arm can move to the joint positions corresponding to the predicted motion space parameters, and the forward and inverse kinematic equations having no solution means that the robotic arm can move to the joint positions corresponding to the predicted motion space parameters.

[0176] In step S8, the robotic arm is controlled to move according to the joint parameters T1, and the environmental information t is updated to t1.

[0177] Adjust the joint parameters of the robotic arm to t1, and obtain the updated environmental parameters with reference to the method in step S1.

[0178] In step S9, through the action determination module, according to the sensor for obtaining the environmental information t1, the visual observation value is obtained, and according to the action recognition model, it is judged whether the action is completed.

[0179] For example, based on the action determination module, visual observations are obtained from the sensors of the environmental information t1 to determine whether the action is completed.

[0180] The process of model training is as follows: construct pictures of the completed states of atomic tasks and the corresponding atomic task encodings, construct a training set, and train a small vision model or fine-tune a large vision model so that it can recognize whether an atomic task is completed.

[0181] In step S10, if the action is completed, the observations Di, Ri, and the pose prediction Wi at each moment i from the start to the completion of the atomic task, and the reward value Q(i) are sequentially passed to the action model.

[0182] Among them, Di, Ri, and Wi are the depth information, visual features, and pose at the i-th moment respectively, and Q(i) is the reward value at the i-th moment. The reward value is obtained by multiplying the maximum excitation value by the decay coefficient. The maximum excitation value can be set according to user requirements. For example, it can be set to 10000, 1000, or other values.

[0183] For example, a robotic arm reinforcement learning training method that determines whether an atomic task is completed through a large or small vision model for visual recognition, thereby determining the reward function.

[0184] In step S11, the completion status information of whether the atomic task is completed is obtained through the instruction semantics module, and the context of the atomic task is updated according to the completion status information.

[0185] In the case where the atomic task is completed, the atomic task to be executed is updated to the next atomic task in the atomic task list.

[0186] In step S12, it is judged whether the execution times of the current atomic task reach the set threshold K. If not, return to step S2. If the set threshold K is reached, or the forward and inverse kinematic equations have no solution, enter step S13.

[0187] Among them, the set threshold K can be set according to user requirements. For example, if the atomic task is to grasp an object from position A and the set threshold is 20, the action of the robotic arm from the current position to grasping the object at position A can be decomposed into 20 sub-tasks. Reaching the set threshold K, or the forward and inverse kinematic equations having no solution, indicates that the task fails.

[0188] In step S13, the reward value is set to -Q(i), step S10 is executed, and then return to step S1.

[0189] The action model is trained until the model converges for each atomic task. The action model is used to explore and learn new actions until the model converges.

[0190] In the new open scenario, cameras are installed at the same position of the robotic arm and at fixed positions in the environment with no obvious dead angles, and the trained robotic arm control model obtained in the embodiment can be used for robotic arm motion planning and verification.

[0191] In this embodiment, through pre-training the visual model, tasks in an open multi-task scenario can be decomposed. Through training the visual model, scene expansion can be performed. Through the 3D depth algorithm, grasping and obstacle avoidance in the open scenario can be achieved. The model is cascaded through the reinforcement learning algorithm and the pose calculation module to realize self-supervised training of the multi-task open scenario grasping task without action annotation. In this embodiment, through task decomposition, depth information acquisition, and reward function design, self-supervised reinforcement learning training without action annotation can be realized, reducing the annotation cost compared with the methods of the related art.

[0192] Exemplarily, the robotic arm is a robotic arm including 7 robotic arm joints. During the process of reinforcement learning, according to the physical characteristics of the 7-axis robotic arm, the action space of the action model is improved. Specifically, the rotation angle of the upper arm around the shoulder-wrist joint connection line is added based on the end pose of the robotic arm. By restricting the opening and closing angle of joint 4 through the pose calculation module, a unique mapping from the action space to the robotic arm joint angle can be realized on the premise that the action space parameters are reachable, avoiding the situation of infinite solutions when mapping the end pose of the 7-axis robotic arm to joint parameters and the existence of redundant robotic arms, and avoiding the problem of non-linear joint angle control to reach the target pose, improving the convergence of the model.

[0193] Exemplarily, the robotic arm control method provided in the embodiment of the present application may include the following steps:

[0194] Step F1, initialize the context of the atomic tasks in the instruction semantics module, initialize the environmental information, obtain the environmental information t at the current moment, and set the task to be executed.

[0195] Obtain the task instruction of the robotic arm, initialize the context information of the task instruction, initialize the environmental information of the operation area, obtain the environmental information at the current moment, and set the task that the robotic arm is currently to execute.

[0196] Step F2, through the instruction semantics module, according to the task instruction and the context status information of the task instruction, select the currently needed and unfinished atomic task from the list of atomic tasks corresponding to the task to be executed currently, and output the semantic encoding E of the atomic task.

[0197] The method of this step can refer to the description of step S2 and will not be elaborated here.

[0198] Step F3: Obtain the depth information D and visual features R corresponding to the environmental information t through the environmental perception module and according to the environmental information t obtained by the sensor.

[0199] For the method of this step, reference can be made to the description of the aforementioned step S3, and details will not be repeated here.

[0200] Step F4: Obtain the current joint parameters T of each joint of the robotic arm through the pose calculation module, and obtain the pose parameters W of the robotic arm based on the forward and inverse kinematic equations.

[0201] Among them, W includes the end position (x, y, z), end pose (Y, P, R), and the angle r of the upper arm rotating around the shoulder-wrist connection line.

[0202] For the method of this step, reference can be made to the description of the aforementioned step S4, and details will not be repeated here.

[0203] Step F5: Obtain the semantic encoding E, depth information D, visual features R, and the pose parameters W of the robotic arm through the action model, and obtain the predicted action space parameters W1 of the robotic arm based on the pose prediction model.

[0204] Among them, the end position (x1, y1, z1), end pose (Y1, P1, R1), and the angle r1 of the upper arm rotating around the shoulder-wrist connection line.

[0205] For the method of this step, reference can be made to the description of the aforementioned step S5, and details will not be repeated here.

[0206] Step F6: Obtain the predicted action space parameters W1 through the pose calculation module, and calculate the joint parameters T1 of each joint of the robotic arm through the forward and inverse kinematic equations with the movement range of the elbow joint opening and closing as the constraint.

[0207] For the method of this step, reference can be made to the description of the aforementioned step S6, and details will not be repeated here.

[0208] Step F7: Control the movement of the robotic arm according to the joint parameters T1, update the environmental information T to T1, and then return to Step F2.

[0209] For the method of this step, reference can be made to the description of the aforementioned step S7, and details will not be repeated here.

[0210] Refer to Figure 7 , this embodiment of the present application also provides a robotic arm control model training device. The robotic arm includes at least one robotic arm joint. The device 40 includes:

[0211] The first acquisition module 401 is configured to acquire the current atomic task sample of the robotic arm, the current action space parameter sample, and the current visual environment parameter sample of the working area of the robotic arm; the atomic task sample is obtained by decomposing the target task sample to be executed by the robotic arm;

[0212] The second acquisition module 402 is configured to acquire the predicted action space parameter sample of the robotic arm according to the current atomic task sample, the current action space parameter sample, the current visual environment parameter sample, and the robotic arm control model;

[0213] The third acquisition module 403 is configured to control the operation of the robotic arm according to the predicted action space parameter sample, and acquire the updated visual environment parameter sample of the working area of the robotic arm after the operation;

[0214] The fourth acquisition module 404 is configured to obtain the execution result according to the updated visual environment information and the preset execution result judgment model, and train the robotic arm control model according to the execution result to obtain the trained robotic arm control model.

[0215] Exemplarily, the second acquisition module 402 may include:

[0216] The first acquisition sub-module is configured to input the atomic task sample into a preset semantic model to obtain the semantic encoding sample of the atomic task sample; the semantic encoding sample is used to reflect the semantic content of the atomic task sample;

[0217] The second acquisition sub-module is configured to obtain the predicted action space parameter sample of the robotic arm according to the semantic encoding sample, the current action space parameter sample, the current visual environment parameter, and the robotic arm control model.

[0218] Optionally, the robotic arm includes seven robotic arm joints, and the third acquisition module 403 may include:

[0219] The third acquisition sub-module is configured to obtain the preset activity range of the target joint among the robotic arm joints;

[0220] The fourth acquisition sub-module is configured to use the preset activity range as a constraint condition, and obtain the joint parameter sample of the robotic arm joint according to the predicted action space parameter sample and the preset joint parameter calculation model;

[0221] The control sub-module is configured to control the operation of the robotic arm according to the joint parameter sample.

[0222] Optionally, the predicted action space parameter sample includes at least one of the following: the pose prediction value of the end of the robotic arm, and the rotation angle prediction value of the robotic arm's upper arm around the shoulder-wrist connection line.

[0223] Optionally, the third acquisition module 403 includes:

[0224] The fifth acquisition sub-module is used to acquire the current joint parameter sample of the robotic arm joint;

[0225] The sixth acquisition sub-module is used to input the current joint parameter sample into a preset action space parameter calculation model to obtain the current action space parameter sample; wherein, the current action space parameter sample includes the pose information of the end of the robotic arm and the rotation angle of the robotic arm's upper arm around the shoulder-wrist connection line.

[0226] Optionally, the atomic task includes multiple decomposition actions to be executed sequentially. The second acquisition module 402 includes:

[0227] The seventh acquisition sub-module is used to obtain the current predicted action space parameter sample of the robotic arm according to the current decomposition action, the current action space parameter sample, the current visual environment parameter sample, and the robotic arm control model.

[0228] The third acquisition module 403 includes:

[0229] The eighth acquisition sub-module is used to obtain the number of executed decomposition actions of the executed decomposition actions;

[0230] The first determination sub-module is used to determine that the execution result is task completion when the number of executed actions is less than or equal to the total number of all decomposition actions and it is determined that the robotic arm has completed the current atomic task according to the updated visual environment information and the preset execution result judgment model.

[0231] The second determination sub-module is used to determine that the execution result is task not completed when the number of executed actions is greater than the total number of all decomposition actions or the robotic arm cannot reach the position corresponding to the current predicted action space parameter sample.

[0232] Optionally, the fourth acquisition module 404 includes:

[0233] The ninth acquisition sub-module is used to acquire the predicted action space parameter sample, environment information, and reward value corresponding to each executed decomposition action when the execution result is task completion;

[0234] The tenth acquisition sub-module is used to train the robotic arm control model according to the predicted action space parameter sample and the reward value corresponding to each decomposition action to obtain the trained robotic arm control model.

[0235] Optionally, the tenth acquisition sub-module includes:

[0236] The first acquisition unit is used to acquire the execution order of the decomposition action among the multiple decomposition actions of the multiple atomic tasks;

[0237] The first determination unit is used to determine the attenuation coefficient of the decomposition action according to the execution order;

[0238] A second acquisition unit, configured to acquire the product of the attenuation coefficient and a preset reward value, and determine the product as the reward value of the decomposition action.

[0239] In this embodiment, by predicting the predicted action space parameter samples of the robotic arm and training the model according to the predicted action space parameter samples, compared with the method of joint angle prediction in related technologies, the non-linear joint angle prediction is avoided, which is convenient for the model to converge. By decomposing the target task samples, atomic task samples are obtained, and the robotic arm control model is trained separately for each atomic task sample, so that the obtained robotic arm control model has high accuracy. This embodiment does not need to obtain a large amount of manual operation data and use the manual operation data as samples to train the model. Therefore, the acquisition step of training samples is saved, the data processing amount is reduced, and the model training efficiency is improved.

[0240] Referring to Figure 8 , this embodiment also provides a robotic arm control device. The robotic arm includes at least one robotic arm joint. The device 50 includes:

[0241] A fifth acquisition module 501, configured to acquire the current atomic task of the robotic arm, the current pose information, and the current environment information of the robotic arm operation area; the atomic task is a task obtained by decomposing the target task, and the target task sample is used to describe the task to be executed by the robotic arm;

[0242] A sixth acquisition module 502, configured to obtain the predicted action space parameters of the robotic arm according to the atomic task, the current pose information, the current environment information, and the trained robotic arm control model;

[0243] A seventh acquisition module 503, configured to control the operation of the robotic arm according to the predicted action space parameters; wherein, the robotic arm control model is obtained according to the robotic arm control model training method in any one of the above embodiments.

[0244] In this embodiment, by predicting the predicted action space parameter samples of the robotic arm and training the model according to the predicted action space parameter samples, compared with the method of joint angle prediction in related technologies, the non-linear joint angle prediction is avoided, which is convenient for the model to converge. By decomposing the target task samples, atomic task samples are obtained, and the robotic arm control model is trained separately for each atomic task sample, so that the obtained robotic arm control model has high accuracy. Based on this robotic arm control model, the pose parameters of the robotic arm are predicted, and based on the pose parameters, the operation of the robotic arm can be accurately controlled.

[0245] The present invention also provides an electronic device. Refer to Figure 9, including: a processor 601, a memory 602, and a computer program 6021 stored in the memory and executable on the processor. When the processor executes the program, it implements the robotic arm control model training method or the robotic arm control method of the foregoing embodiments.

[0246] The present invention also provides a readable storage medium. When the instructions in the storage medium are executed by the processor of an electronic device, the electronic device can execute the robotic arm control model training method or the robotic arm control method of the foregoing embodiments.

[0247] The present invention also provides a robotic arm, including the robotic arm control model training device, the robotic arm control device, or the electronic device of the foregoing embodiments.

[0248] For the device embodiments, since they are basically similar to the method embodiments, they are described relatively simply. For the related parts, refer to the partial description of the method embodiments.

[0249] It should be noted that all kinds of information and data obtained in the embodiments of the present invention are obtained under the authorization of the information / data holder. The algorithms and displays provided here are not inherently related to any specific computer, virtual system, or other device. Various general systems can also be used in conjunction with the teachings provided here. The structure required to construct such systems is obvious from the above description. In addition, the present invention is not directed to any specific programming language. It should be understood that the content of the present invention described here can be implemented using various programming languages, and the description of the specific language above is to disclose the best implementation mode of the present invention.

[0250] In the specification provided here, a large number of specific details are described. However, it can be understood that the embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures, and technologies are not shown in detail so as not to obscure the understanding of this specification. Similarly, it should be understood that, in order to streamline the present invention and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be construed as reflecting the intention that the claimed invention requires more features than those expressly recited in each claim. Rather, as reflected in the following claims, the inventive aspects lie in less than all the features of the single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into the detailed description, where each claim stands on its own as a separate embodiment of the present invention.

[0251] Those skilled in the art can understand that the modules in the devices in the embodiments can be adaptively changed and set in one or more devices different from the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and in addition, they can be divided into multiple sub-modules or sub-units or sub-components. Except that at least some of such features and / or processes or units are mutually exclusive, any combination can be adopted to combine all the features disclosed in this specification (including the accompanying claims, abstract and drawings) and all the processes or units of any method or device so disclosed. Unless otherwise explicitly stated, each feature disclosed in this specification (including the accompanying claims, abstract and drawings) can be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0252] Each component embodiment of the present invention can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components in the sorting device according to the present invention. The present invention can also be implemented as a device or device program for executing part or all of the methods described herein. Such a program for implementing the present invention can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0253] It should be noted that the above embodiments illustrate the present invention rather than limit the present invention, and those skilled in the art can design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention can be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In the unit claims listing several devices, several of these devices can be embodied by the same hardware item. The use of the words first, second, and third, etc. does not denote any order. These words can be interpreted as names.

[0254] The user information involved in the present invention (including but not limited to the user's device information, user personal information, etc.), relevant data, etc. are all information authorized by the user or authorized by all parties.

[0255] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein. The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention. The above are only the specific implementation manners of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or replacements, which should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for training a robotic arm control model, characterized in that, Including: Obtaining the current atomic task sample of the robotic arm, the current action space parameter sample, and the current visual environment parameter sample of the working area of the robotic arm; The atomic task sample is obtained by decomposing the target task sample to be executed by the robotic arm; According to the current atomic task sample, the current action space parameter sample, the current visual environment parameter sample, and the robotic arm control model, obtaining the predicted action space parameter sample of the robotic arm; The predicted action space parameter sample includes: the predicted pose value of the end of the robotic arm and the predicted rotation angle value of the upper arm of the robotic arm around the shoulder-wrist connection line; Controlling the operation of the robotic arm according to the predicted action space parameter sample, and obtaining the updated visual environment parameter sample of the working area of the robotic arm after operation; Obtaining the execution result according to the updated visual environment information and the preset execution result judgment model, and training the robotic arm control model according to the execution result to obtain the trained robotic arm control model.

2. The method according to claim 1, characterized in that, The obtaining the predicted action space parameter sample of the robotic arm according to the current atomic task sample, the current action space parameter sample, the current visual environment parameter sample, and the robotic arm control model includes: Inputting the atomic task sample into a preset semantic model to obtain the semantic encoding sample of the atomic task sample; the semantic encoding sample is used to reflect the semantic content of the atomic task sample; According to the semantic encoding sample, the current action space parameter sample, the current visual environment parameter, and the robotic arm control model, obtaining the predicted action space parameter sample of the robotic arm.

3. The method according to claim 1, wherein The robotic arm includes seven robotic arm joints. The controlling the operation of the robotic arm according to the predicted action space parameter sample includes: Obtaining the preset activity range of the target joint among the robotic arm joints; Taking the preset activity range as a constraint condition, and according to the predicted action space parameter sample and a preset joint parameter calculation model, obtaining the joint parameter sample of the robotic arm joints; Controlling the operation of the robotic arm according to the joint parameter sample.

4. The method according to claim 1, characterized in that, Obtaining the current action space parameter sample of the robotic arm includes: Obtaining the current joint parameter sample of the robotic arm joints; Inputting the current joint parameter sample into a preset action space parameter calculation model to obtain the current action space parameter sample; Wherein, the current action space parameter sample includes the pose information of the end of the robotic arm and the rotation angle of the upper arm of the robotic arm around the shoulder-wrist connection line.

5. The method according to claim 1, wherein The atomic task includes multiple decomposition actions to be executed in sequence. The obtaining the predicted action space parameter sample of the robotic arm according to the current atomic task sample, the current action space parameter sample, the current visual environment parameter sample, and the robotic arm control model includes: According to the current decomposition action, the current action space parameter sample, the current visual environment parameter sample, and the robotic arm control model, obtaining the current predicted action space parameter sample of the robotic arm.

6. The method according to claim 5, wherein Obtaining the execution result according to the updated visual environment information and the preset execution result judgment model includes: Obtaining the executed number of the executed decomposition actions; When the number of executed actions is less than or equal to the total number of all decomposed actions, and it is determined that the robotic arm has completed the current atomic task according to the updated visual environment information and the preset execution result judgment model, determine that the execution result is task completed; When the number of executed actions is greater than the total number of all decomposed actions, or the robotic arm cannot reach the position corresponding to the current predicted action space parameter sample, determine that the execution result is task not completed.

7. The method according to claim 5, wherein Training the robotic arm control model according to the execution result to obtain a trained robotic arm control model, including: When the execution result is task completed, obtain the predicted action space parameter sample, environment information, and reward value corresponding to each executed decomposed action; Train the robotic arm control model according to the predicted action space parameter sample and reward value corresponding to each decomposed action to obtain a trained robotic arm control model.

8. The method according to claim 7, wherein Obtaining the reward value of the decomposed action includes: Obtain the execution order of the decomposed action among the multiple decomposed actions of the atomic task; Determine the attenuation coefficient of the decomposed action according to the execution order; Obtain the product of the attenuation coefficient and the preset reward value, and determine the product as the reward value of the decomposed action.

9. A manipulator control method, characterized in that, Including: Obtain the current atomic task of the robotic arm, the current action space parameter, and the current visual environment parameter of the robotic arm working area; The atomic task is obtained by decomposing the target task to be executed by the robotic arm; According to the atomic task, the current action space parameter, the current visual environment parameter, and the trained robotic arm control model, obtain the predicted action space parameter of the robotic arm; Control the operation of the robotic arm according to the predicted action space parameter; the robotic arm control model is obtained by the robotic arm control model training method according to any one of claims 1 to 8.

10. An apparatus for training a robotic arm control model, characterized in that, The device includes: The first acquisition module is used to acquire the current atomic task sample of the robotic arm, the current action space parameter sample, and the current visual environment parameter sample of the robotic arm working area; the atomic task sample is obtained by decomposing the target task sample to be executed by the robotic arm; The second acquisition module is used to obtain the predicted action space parameter sample of the robotic arm according to the current atomic task sample, the current action space parameter sample, the current visual environment parameter sample, and the robotic arm control model; the predicted action space parameter sample includes: the pose prediction value of the end of the robotic arm and the rotation angle prediction value of the robotic arm's upper arm around the shoulder-wrist connection line; The third acquisition module is used to control the operation of the robotic arm according to the predicted action space parameter sample and acquire the updated visual environment parameter sample of the robotic arm working area after operation; The fourth acquisition module is used to obtain the execution result according to the updated visual environment information and the preset execution result judgment model, and train the robotic arm control model according to the execution result to obtain a trained robotic arm control model.

11. A robotic arm control device, characterized in that, The device includes: A fifth acquisition module, configured to acquire the current atomic task of the robotic arm, the current action space parameters, and the current visual environment parameters of the working area of the robotic arm; the atomic task is obtained by decomposing the target task to be executed by the robotic arm. A sixth acquisition module, configured to obtain the predicted action space parameters of the robotic arm according to the atomic task, the current action space parameters, the current visual environment parameters, and the trained robotic arm control model. A seventh acquisition module, configured to control the operation of the robotic arm according to the predicted action space parameters; the robotic arm control model is obtained by the robotic arm control model training method according to any one of claims 1 to 8.

12. An electronic device, characterized in that, Comprising: A processor, a memory, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the program, it implements the robotic arm control model training method according to any one of claims 1-8 or the robotic arm control method according to claim 9.

13. A readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is enabled to execute the robotic arm control model training method according to any one of claims 1-8 or the robotic arm control method according to claim 9.

14. A robotic arm, characterized in that, The robotic arm includes the robotic arm control model training device according to claim 10, the robotic arm control device according to claim 11, or the electronic device according to claim 12.

Images