Mechanical arm control model training method, mechanical arm control method, equipment and medium

Abstract

The embodiment of the invention provides a mechanical arm control model training method, a mechanical arm control method, equipment and a medium. The method comprises the steps that a current atomic task sample and a current action space parameter sample of the mechanical arm and a current visual environment parameter sample of a mechanical arm operation area are obtained; according to the current atomic task sample, the current action space parameter sample, the current visual environment parameter sample and a mechanical arm control model, a predicted action space parameter sample of the mechanical arm is obtained, the mechanical arm is controlled to operate according to the predicted action space parameter sample, and an updated visual environment parameter sample of a mechanical arm operation area after operation is obtained; and an execution result is obtained according to the updated visual environment information and a preset execution result judgment model, a mechanical arm control model is trained according to the execution result, and a trained mechanical arm control model is obtained. The training efficiency and accuracy of the mechanical arm control model are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot arm control, and in particular relates to a robot arm control model training method, a robot arm control method, a device and a medium. Background Art

[0002] Robotic arms have always found it difficult to work in open and complex scenarios like humans. With the development of reinforcement learning technology in artificial intelligence, it has become possible for robotic arms to think and act like humans.

[0003] In related technologies, based on supervised reinforcement learning methods, a large amount of manual operation data can be obtained through VR and other devices, a training set can be built based on the manual operation data, and the robotic arm can learn manual actions based on the manual operation data. However, this method has high data acquisition costs and low efficiency, so it is difficult to expand to open scenarios.

[0004] In related technologies, common unsupervised reinforcement learning methods usually learn the optimal strategy through random trials or fixed strategy trials. This method requires a lot of training time and large data processing volume, resulting in high learning costs for the robot control strategy, and the robot control model is difficult to converge in complex scenarios. Summary of the invention

[0005] The present invention provides an unsupervised robotic arm control model training method, a robotic arm control method, a device and a medium based on task semantic decomposition and reinforcement learning technology, so as to solve the problems of high training cost and poor convergence of the robotic arm control model in the related technology.

[0006] In a first aspect, the present invention provides a method for training a robot arm control model, comprising:

[0007] Obtaining the current atomic task sample of the robot arm, the current action space parameter sample, and the current visual environment parameter sample of the robot arm's operating area; the atomic task sample is obtained by decomposing the target task sample to be executed by the robot arm;

[0008] Acquire a predicted action space parameter sample of the robotic arm according to the current atomic task sample, the current action space parameter sample, the current visual environment parameter sample, and the robotic arm control model;

[0009] Controlling the operation of the robot arm according to the predicted action space parameter samples, and obtaining updated visual environment parameter samples of the operating area of ​​the robot arm after the operation;

[0010] The execution result is obtained according to the updated visual environment information and the preset execution result judgment model, and the robotic arm control model is trained according to the execution result to obtain a trained robotic arm control model.

[0011] In a second aspect, the present invention provides a robot arm control method, comprising:

[0012] Obtaining the current atomic task of the robot, the current action space parameters, and the current visual environment parameters of the robot's operating area; the atomic task is obtained by decomposing the target task to be performed by the robot;

[0013] Obtaining predicted motion space parameters of the robotic arm according to the atomic task, the current motion space parameters, the current visual environment parameters, and the trained robotic arm control model;

[0014] The operation of the robotic arm is controlled according to the predicted motion space parameters; the robotic arm control model is obtained according to the robotic arm control model training method described in the first aspect.

[0015] In a third aspect, the present invention provides a robot arm control model training device, the device comprising:

[0016] The first acquisition module is used to acquire the current atomic task sample of the robot arm, the current action space parameter sample, and the current visual environment parameter sample of the robot arm's operating area; the atomic task sample is obtained by decomposing the target task sample to be executed by the robot arm;

[0017] A second acquisition module is used to acquire a predicted action space parameter sample of the manipulator according to the current atomic task sample, the current action space parameter sample, the current visual environment parameter sample, and the manipulator control model;

[0018] A third acquisition module is used to control the operation of the robot arm according to the predicted action space parameter samples, and obtain updated visual environment parameter samples of the operating area of ​​the robot arm after the operation;

[0019] The fourth acquisition module is used to obtain the execution result according to the updated visual environment information and the preset execution result judgment model, and train the robotic arm control model according to the execution result to obtain the trained robotic arm control model.

[0020] In a fourth aspect, the present invention provides a robot arm control device, the device comprising:

[0021] A fifth acquisition module is used to acquire the current atomic task of the robot arm, the current action space parameters, and the current visual environment parameters of the robot arm's operating area; the atomic task is obtained by decomposing the target task to be performed by the robot arm;

[0022] a sixth acquisition module, configured to obtain predicted motion space parameters of the robotic arm according to the atomic task, the current motion space parameters, the current visual environment parameters, and the trained robotic arm control model;

[0023] A seventh acquisition module is used to control the operation of the robotic arm according to the predicted action space parameters; the robotic arm control model is obtained according to the robotic arm control model training method described in any one of the first aspects.

[0024] In a fifth aspect, the present invention provides an electronic device comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the robotic arm control model training method described in the first aspect or the robotic arm control method described in the second aspect is implemented.

[0025] In a sixth aspect, the present invention provides a readable storage medium. When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the robotic arm control model training method described in the first aspect, or the robotic arm control method described in the second aspect.

[0026] In summary, by predicting the predicted action space parameter samples of the robot arm and training the model according to the predicted action space parameter samples, compared with the joint angle prediction method in the related art, nonlinear joint angle prediction is avoided, which facilitates model convergence. By decomposing the target task samples, atomic task samples are obtained, and the robot arm control model is trained for each atomic task sample, the obtained robot arm control model has high accuracy. This embodiment does not need to obtain a large amount of manual operation data, and uses the manual operation data as a sample training model, thereby saving the step of obtaining training samples, reducing the amount of data processing, and improving the efficiency of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 is a flowchart of the steps of a robot arm control model training method provided by an embodiment of the present invention;

[0029] Figure 2 is a schematic diagram of another mechanical arm provided by an embodiment of the present invention;

[0030] Figure 3 is a flowchart of the steps of a robot arm control model training method provided by an embodiment of the present invention;

[0031] Figure 4 is a flowchart of a method for controlling a robotic arm provided by an embodiment of the present invention;

[0032] Figure 5 is a flowchart of the steps of a robot arm control model training method provided by an embodiment of the present invention;

[0033] Figure 6 is a flowchart of the steps of a robot arm control model training method provided by an embodiment of the present invention;

[0034] Figure 7 is a structural diagram of a robot arm control model training device provided by an embodiment of the present invention;

[0035] Figure 8 is a structural diagram of a robot arm control device provided by an embodiment of the present invention;

[0036] Fig. 9 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] The following is a further description of the robot arm control model training method of the present application in conjunction with the relevant drawings and embodiments:

[0039] Figure 1 is a flowchart of a method for training a robot arm control model provided by an embodiment of the present invention. Figure 1 As shown, the method may include:

[0040] Step 101, obtaining the current atomic task sample of the robot arm, the current action space parameter sample, and the current visual environment parameter sample of the robot arm working area.

[0041] Among them, the atomic task samples are obtained by decomposing the target task samples to be performed by the robotic arm.

[0042] Specifically, the atomic task sample is the semantic encoding of the task obtained by decomposing the target task sample, and the target task sample is used to describe the task sample to be executed by the robot arm.

[0043] For example, if the task to be performed by the robotic arm is to move an object from position A to position B, the task to be performed can be decomposed to obtain the following atomic tasks: the robotic arm moves from the initial position to position A and grasps the object; the robotic arm lifts the object and places it at position B.

[0044] The visual environment parameter samples include depth information and visual features of the working area. For example, the 3D environment information of the working area of ​​the robot arm can be constructed through a 3D reconstruction algorithm, and the 3D environment information includes depth information, and the visual features can be obtained through the trained visual recognition algorithm and depth information.

[0045] The motion space parameter sample is a set of the robot arm motion parameters in a given working environment. The motion space parameter sample reflects the motion parameters of the robot arm, for example, it may include posture parameters and the predicted value of the rotation angle of the robot arm around the shoulder-wrist connection line.

[0046] Step 102 , obtaining a predicted action space parameter sample of the robot arm according to the current atomic task sample, the current action space parameter sample, the current visual environment parameter sample, and the robot arm control model.

[0047] For example, the robot control model may include an action prediction model, wherein the action space parameter calculation model may be a Deep Deterministic Policy Gradient (DDPG) model or an Asynchronous Advantage Actor-Critic (A3C) model. For example, the current atomic task sample, the current action space parameter sample, and the current visual environment parameter sample are input into the action prediction model to obtain the predicted action space parameter sample of the robot.

[0048] Specifically, the atomic task samples, the current action space parameter samples, and the current visual environment parameter samples are input into the robot arm control model to obtain the predicted action space parameters predicted by the robot arm control model.

[0049] Step 103, controlling the operation of the robot arm according to the predicted action space parameter samples, and obtaining updated visual environment parameter samples of the operating area of ​​the robot arm after the operation.

[0050] Furthermore, an action sequence of joint rotation parameters is obtained according to the predicted action space parameter samples, and the operation of the robot arm is controlled based on the action sequence of the joint rotation parameters.

[0051] For example, the rotation parameters of each joint of the robot arm are calculated according to the predicted action space parameters, the action sequence of the joint rotation parameters of the robot arm is constructed according to the rotation parameters of each joint, and the operation of the robot arm is controlled according to the action sequence of the joint rotation parameters. Figure 2 , the joint rotation parameter is the rotation angle of the robot joint. Figure 2 In the figure, r represents the rotation angle of the robot joint, ∑0 is the horizontal plane coordinate system, and ∑7 is the reference coordinate system of the 7th robot joint.

[0052] The robot arm operates according to the predicted action space parameters. After the operation, the visual environment captured by the sensors on the robot arm will also change. Correspondingly, the visual environment parameter samples of the robot arm's working area after the operation will also change, from the current visual environment parameter samples in step 101 to the updated visual environment parameter samples.

[0053] Step 104, obtaining the execution result according to the updated visual environment information and the preset execution result judgment model, and training the robotic arm control model according to the execution result to obtain a trained robotic arm control model.

[0054] The execution result includes whether the task is completed or not. For example, the atomic task described by the atomic task sample is that the robot arm grasps the object at position A. After the robot arm runs according to the predicted action space parameter sample, if the robot arm grasps the object from position A, it indicates that the execution result is that the task is completed; otherwise, the execution result is that the task is not completed.

[0055] For example, updating the visual environment information may include depth information and visual features in the updated visual environment after the robotic arm operates according to the current action space parameter sample.

[0056] For example, the preset execution result judgment model can be a convolutional neural network model (YouOnly LookOnce, YOLO).

[0057] For example, according to the correspondence between the preset execution result and the preset reward value, the reward value corresponding to the execution result is obtained, and the robotic arm control model is trained according to the reward value to obtain the trained robotic arm control model.

[0058] With the development of big model technology, various applications based on big models are emerging. Embodied intelligence is the main application direction of big models. In the field of embodied intelligence, how to operate the robot arm in open and complex scenarios has always been a hot topic and a difficult problem in this field.

[0059] In this embodiment, a sample of predicted motion space parameters of the robot arm is obtained, and a model is trained based on the sample of predicted motion space parameters. Compared with the method of training the model by predicting joint angles in the related art, this embodiment avoids nonlinear joint angle prediction and facilitates model convergence. By decomposing the target task sample, an atomic task sample is obtained, and the robot arm control model is trained for each atomic task sample, the obtained robot arm control model has high accuracy. This embodiment does not need to obtain a large amount of manual operation data, and uses the manual operation data as a sample training model, thereby saving the step of obtaining training samples, reducing the amount of data processing, and improving the efficiency of model training.

[0060] In addition, this embodiment is suitable for robot arm operations in open scenarios. By decomposing the complex grasping tasks of the robot arm in open scenarios into atomic tasks and training the model for the atomic tasks, it avoids designing reward functions and state spaces separately for each grasping task of the robot arm, avoids repetitive operation steps in model design, and improves the generalization and extensibility of the robot arm control model.

[0061] Reference Figure 3 , the robot arm control model training method may include the following steps:

[0062] Step 201, obtaining a sample of the current visual environment parameters of the robot arm operation area.

[0063] Among them, the atomic task samples are obtained by decomposing the target task samples to be performed by the robotic arm.

[0064] The method of this step has been described in the aforementioned step 101 and will not be repeated here.

[0065] Step 202, obtaining current joint parameter samples of the robot arm joints.

[0066] The current joint parameter sample may include the rotation angle of each joint of the robotic arm relative to the initial state in the current state of the robotic arm.

[0067] Step 203: input the current joint parameter sample into a preset motion space parameter calculation model to obtain the current motion space parameter sample.

[0068] Among them, the current action space parameter sample includes the posture information of the end of the robot arm and the rotation angle of the robot arm around the shoulder-wrist connection line.

[0069] In this step, the current motion space parameter sample includes the rotation angle of the upper arm of the robot around the shoulder-wrist connection line. On the basis of the coordinate information and angle information of the end of the robot, the rotation angle of the upper arm around the shoulder-wrist joint connection line is added to form the motion space parameter sample in this embodiment. Subsequently, through the current motion space parameter sample and the preset range of motion of the target joint, the unique mapping of the motion space and the joint angle of the robot arm can be achieved under the premise that the motion space is reachable, avoiding the situation where there are infinite solutions for redundant robots when mapping joint parameters of the end posture of the multi-axis (for example, 7-axis) robot arm. At the same time, it can simplify the problem of nonlinear joint angle control in order to achieve the target posture, which is convenient for the convergence of the robot control model.

[0070] Step 204: input the atomic task sample into a preset semantic model to obtain a semantic coding sample of the atomic task sample.

[0071] Among them, the semantic coding samples are used to reflect the semantic content of the atomic task samples.

[0072] Among them, the semantic model can be a bidirectional encoder model based on Transform (BidirectionalEncoder Representations from Transformers, BERT), or a large language model (LargeLanguage Model, LLM), or other semantic models.

[0073] Step 205 , obtaining a predicted action space parameter sample of the robotic arm according to the semantic coding sample, the current action space parameter sample, the current visual environment parameter, and the robotic arm control model.

[0074] The predicted motion space parameter samples include at least one of the following: a predicted value of the posture of the end of the robotic arm, and a predicted value of the rotation angle of the robotic arm around the shoulder-wrist connection line.

[0075] Specifically, the semantic coding samples, the current action space parameter samples, and the current visual environment parameters are input into the robot control model to obtain the predicted action space parameter samples of the robot.

[0076] Step 206, obtaining a preset range of motion of a target joint in the robot arm joint.

[0077] For example, refer to Figure 2 ,The robotic arm includes 7 robotic arm joints, among which the elbow joint is determined as the target joint, and the preset activity range of the target joint is set to 0 to 180 degrees.

[0078] Step 207 , taking the preset range of motion as a constraint condition, obtaining joint parameter samples of the robot arm joints according to the predicted motion space parameter samples and the preset joint parameter calculation model.

[0079] For example, the preset joint parameter calculation model can be a forward and inverse kinematics equation. The predicted motion space parameter samples are used as input parameters of the forward and inverse kinematics equations, and the preset range of motion is used as a constraint condition to obtain the joint parameters calculated by the forward and inverse kinematics equations.

[0080] Among them, the predicted action space parameter samples include the posture parameters of the end of the robotic arm. The robotic arm includes multiple robotic arm joints. Substituting the predicted action space samples into the forward and inverse kinematics equations, the joint parameters of each robotic arm joint can be obtained.

[0081] Step 208, controlling the operation of the robot arm according to the joint parameter samples.

[0082] For example, the joint parameter sample includes the rotation angle of each robotic arm joint. According to the rotation angle of each robotic arm joint, the operation of the robotic arm is controlled so that the rotation angle of each robotic arm joint is equal to the rotation angle of each robotic arm joint in the joint parameter sample.

[0083] Taking the preset range of motion of the target joint as a constraint, the joint parameter samples of the robot arm joints are obtained according to the predicted action space parameter samples and the preset joint parameter calculation model. Thus, it can be ensured that the joint parameters included in the obtained joint parameter samples meet the constraints of the preset range of motion of the target joint, and it can be ensured that the joint parameter samples meet the operation requirements of the robot arm.

[0084] Step 209, obtaining updated visual environment parameter samples of the robot arm's operating area after the robot arm is controlled to operate according to the joint parameter samples.

[0085] The updated environment information includes the depth information and visual features of the working area of ​​the robot arm after the robot arm runs according to the joint parameter sample of step 208.

[0086] For example, after the robot arm is controlled to operate, the information collected by the sensor in the robot arm for collecting environmental information also changes, and correspondingly, the visual environment information of the robot arm's operating area also changes. Further, the method of the aforementioned step 101 can be referred to to obtain the updated depth information and visual features of the robot arm's operating area after the robot arm operates.

[0087] Step 210, obtaining the execution result according to the updated visual environment information and the preset execution result judgment model, and training the robotic arm control model according to the execution result to obtain a trained robotic arm control model.

[0088] The execution result includes: whether the atomic task is completed or not completed.

[0089] For example, the preset action recognition model may be a YOLO model. A sample of environmental information of a known execution result may be obtained, the execution result may be used as a label, and the YOLO model may be trained using the sample and the label to obtain the preset action recognition model.

[0090] According to the updated visual environment parameter samples and the preset action recognition model, the execution results of the atomic task can be quickly obtained.

[0091] For example, the atomic task includes multiple decomposed actions that are executed in sequence, and step 205 may include the following sub-steps:

[0092] Sub-step 2051, obtaining the current predicted action space parameter sample of the robot arm according to the current decomposed action, the current action space parameter sample, the current visual environment parameter sample, and the robot arm control model.

[0093] For example, the atomic task is that the robot arm grasps an object at position A from an initial position. This atomic task can be decomposed into multiple decomposed actions.

[0094] According to the current decomposed action, the current action space parameter sample, the current visual environment parameter sample, and the robot arm control model, the current predicted action space parameter sample of the robot arm is obtained. The obtained predicted action space parameter sample is the predicted action space parameter sample for the current decomposed action. After the atomic task to which the decomposed action belongs is executed, the robot arm control model can be trained based on the predicted action space parameter sample of the decomposed action.

[0095] For example, the atomic task can be decomposed into 20 decomposed actions, each of which corresponds to a path for the robot to move.

[0096] Correspondingly, step 210 may include the following sub-steps:

[0097] Sub-step 2101, obtaining the number of executed decomposition actions.

[0098] For example, each time a decomposed action is executed, the number of executed decomposed actions is increased by one to obtain the number of executed actions.

[0099] Sub-step 2102, when the number of executed actions is less than or equal to the total number of all decomposed actions, and based on the updated visual environment information and the preset execution result judgment model, it is determined that the robot arm has completed the current atomic task, the execution result is determined to be task completion.

[0100] When the number of actions executed is less than or equal to the total number of all decomposed actions, and according to the current predicted action space parameters, it is determined that the robot has completed the current atomic task, it means that all decomposed actions have not been executed yet, but the robot has completed the current atomic task, or all decomposed actions have just been executed, and the robot has completed the current atomic task.

[0101] In this case, it means that the robot arm has completed the current atomic task without exceeding the set number of executions (i.e., the total number of decomposed actions), and the operation of the robot arm is in line with expectations. Therefore, in this case, the execution result is determined to be task completion.

[0102] Sub-step 2103, when the number of executions is greater than the total number of all decomposed actions, or the robot arm cannot reach the position corresponding to the current predicted action space parameter sample, it is determined that the execution result is that the task is not completed.

[0103] When the number of executions is greater than the total number of all decomposed actions, it means that the number of executions of the robot arm is greater than the set number of executions (i.e., the total number of decomposed actions). In this case, the operation of the task robot arm does not meet expectations, and the execution result is determined to be that the task is not completed.

[0104] The total number of decomposed actions is equivalent to the upper limit of the number of actions to be performed to complete the atomic task, and the number of actions that have been executed is equivalent to the number of actions that have been executed. According to the number of decomposed actions that have been executed, the execution result of the current atomic task is judged to be task completion or task incomplete. This judgment result can meet the requirements for the number of robot arm operations.

[0105] When the robot arm cannot reach the position corresponding to the current predicted action space parameters, it means that the robot arm cannot complete the action corresponding to the predicted action space parameters. Therefore, the execution result is determined to be task incomplete.

[0106] Correspondingly, step 210 may include the following sub-steps:

[0107] Sub-step 2104, when the execution result is task completion, obtain the predicted action space parameter samples, environmental information and reward values ​​corresponding to each decomposed action that has been executed.

[0108] For example, for each decomposed action, the execution result is obtained according to the method of step 202 to step 210, and the predicted action space parameter sample and environmental information in the process of obtaining the execution result are obtained. Among them, the reward value of each decomposed action is obtained according to the corresponding relationship between the preset execution result and the preset reward value.

[0109] Wherein, obtaining the reward value of the decomposed action may include the following sub-steps:

[0110] Sub-step A1, obtaining the execution order of the decomposed actions in the multiple decomposed actions of the atomic task.

[0111] For example, an atomic task is decomposed to obtain multiple decomposed actions that need to be executed in sequence, wherein each decomposed action has a corresponding execution order.

[0112] Sub-step A2, determining the attenuation coefficient of the decomposed action according to the execution order.

[0113] For example, the attenuation coefficient is positively correlated with the execution order, that is, the greater the execution order, the greater the attenuation coefficient, and conversely, the smaller the execution order, the smaller the attenuation coefficient.

[0114] Sub-step A3, obtaining the product of the attenuation coefficient and the preset reward value, and determining the product as the reward value of the decomposed action.

[0115] Among them, the preset reward value is related to the execution result. Further, when the execution result is that the task is completed, the preset reward value is Q, and Q>0. When the execution result is that the task is not completed, the preset reward value is -Q.

[0116] The preset reward value may be set according to user needs, for example, it may be set to 10000, 1000, or other values.

[0117] For example, the product of the attenuation coefficient related to the execution order of the decomposed actions and the preset reward value is determined as the reward function. The attenuation coefficient is determined according to the execution order of the decomposed actions, and the reward value of the decomposed action is determined according to the product of the attenuation coefficient and the preset reward value, and the reward value can reflect the degree of influence of the decomposed action on the prediction result.

[0118] In one embodiment, the preset reward value is related to the execution result, and the execution result is a result obtained based on visual recognition. The reward function is obtained based on the preset reward value, thereby realizing the use of a reward function based on visual recognition and improving the scalability of the reward function in open scenarios.

[0119] Sub-step 2105 , training the robotic arm control model according to the predicted action space parameter samples and reward values ​​corresponding to each decomposed action, to obtain a trained robotic arm control model.

[0120] The predicted action space parameter samples, environmental information and reward values ​​of each decomposed action are obtained, and the robotic arm control model is trained according to the predicted action space parameter samples and reward values ​​corresponding to each decomposed action, so as to obtain the robotic arm control model, which is a model related to each decomposed action. When the robotic arm control model is subsequently used to control the robotic arm, the action space parameters of each decomposed action can be obtained, and the operation of the robotic arm can be controlled based on the action space parameters of each decomposed action. This control method has the advantage of high control accuracy.

[0121] Correspondingly, step 210 may include the following sub-steps:

[0122] Sub-step 2106, training the robot arm control model according to the execution result of the atomic task, and obtaining a robot arm control model of the trained atomic task.

[0123] For example, for each atomic task, the method from step 201 to step 209 is used to obtain a robot arm control model of the trained atomic task.

[0124] Sub-step 2107, constructing a robotic arm control model for executing the target task based on the robotic arm control models of the multiple atomic tasks that have been trained.

[0125] For example, the entirety of the robot control models of multiple atomic tasks is determined as the robot control model for executing the target task.

[0126] For example, the execution results of the decomposed actions in all atomic tasks, environmental information, and predicted action space parameters can also be obtained, and the robotic arm control model can be trained based on these data to obtain a trained robotic arm control model.

[0127] In the related art, the cutting-edge robot arm control method is the imitation learning method, but the generation of example data is the bottleneck of this method. In the method of the related art, it is necessary to obtain a large amount of manual operation data so that the robot arm can learn the manual operation action and obtain the robot arm control model; a large number of robot action training sets can also be constructed through manual operation, but the slow manual construction of training data will affect the efficiency of model training. In addition, in the related art, the robot arm joint angle and the relative position of the target object and the robot arm are used as the observation space, the robot arm joint angle is used as the action space, and the reward function is calculated according to the distance between the target object and the robot arm. In an embodiment of the present application, environmental information including depth information, as well as the posture parameters of the robot arm and the semantic encoding of the atomic task can be used as the observation space. The robot arm posture parameters are used as the action space, and the reward value corresponding to the execution result judged by the artificial intelligence model is used as the reward function. Based on this information, the robot arm control model can be quickly trained.

[0128] In the related art, the reward function used to train the robot arm control model may include a main line reward, a target approach reward, an obstacle approach penalty, and an obstacle movement penalty. The state space is the distance between the target object and the robot arm, the minimum circumscribed sphere of all obstacles, and the action space is the angle of each joint and the terminal posture calculated by the angle. The model of this method is to predict the joint angle. In this embodiment, the action space does not include the angle of each joint, so the terminal posture of the action space is not calculated by the angle of each joint, but is directly predicted by the intelligent model. Furthermore, the state space of this embodiment is the depth information of the entire environment, the contextual semantic information based on the context, and the posture information, and the reward function of this embodiment is the grasping state of visual recognition. Based on the predicted action space parameter samples, the problem of difficult model convergence caused by nonlinear joint angles can be avoided.

[0129] In the related art, the model can also be constructed by a learning method of imitating real examples, wherein all training data are pre-constructed standard data. The large model used is a visual large model, and the model features are visual segmentation model features, visual text alignment features, etc. In this embodiment, through the self-supervised reinforcement learning method without action annotation, a semantic or multimodal large model can be used, and the features used are deep information features, visual recognition features, atomic task text encoding, etc. for model training, and the model training efficiency is higher.

[0130] Reference Figure 4 The present application also provides a method for controlling a robot arm, the method comprising:

[0131] Step 301, obtaining the current atomic task of the robot arm, the current action space parameters, and the current visual environment parameters of the robot arm's operating area.

[0132] The atomic task is obtained by decomposing the target task to be performed by the robot arm. The method of this step can refer to the description of the aforementioned step 101, which will not be repeated here.

[0133] Step 302, obtaining predicted motion space parameters of the robotic arm according to the atomic task, the current motion space parameters, the current visual environment parameters, and the trained robotic arm control model.

[0134] For example, the atomic task, the current action space parameters, and the current visual environment parameters are input into the trained robotic arm control model to obtain the predicted action space parameters of the robotic arm.

[0135] The motion space parameters may include the coordinate information and angle information of the end of the robotic arm, as well as the rotation angle of the robotic arm around the shoulder-wrist connection line.

[0136] Step 303: Control the operation of the robot arm according to the predicted motion space parameters.

[0137] For example, according to the predicted action parameters, the joint parameters of each joint of the robot arm are obtained, and the robot arm is controlled to operate based on the action sequence of the joint rotation parameters formed by the joint parameters of each joint of the robot arm. The robot arm control model is obtained according to the robot arm control model training method of any of the above embodiments.

[0138] In this embodiment, by predicting the predicted action space parameter samples of the robot arm, the model is trained according to the predicted action space parameter samples. Compared with the joint angle prediction method in the related art, nonlinear joint angle prediction is avoided, which facilitates model convergence. By decomposing the target task samples, atomic task samples are obtained, and the robot arm control model is trained for each atomic task sample, the obtained robot arm control model has high accuracy. Based on the robot arm control model, the posture parameters of the robot arm are predicted, and based on the posture parameters, the operation of the robot arm can be accurately controlled.

[0139] For example, the robotic arm control model training system of the present application includes the following parts: an environment perception module, an action determination module, an instruction semantic module, a posture calculation module, and an action model.

[0140] The action model is used to output the target posture according to the current state characteristics of the robot, the environmental characteristics output by the environment model, and the atomic task encoding of the instruction semantic module, and generate the action parameters of the robot through posture calculation, and control the operation of the robot according to the action parameters through the action model, and then iteratively update the weight of the action model through the reward model. Furthermore, the environmental perception module is used to obtain visual environment parameter information; the action judgment module is equivalent to the execution result judgment model in the aforementioned embodiment, which is used to judge the execution result of the robot's execution task according to the visual environment parameter information; the instruction semantic module is equivalent to the preset semantic model in the aforementioned embodiment, which is used to convert the atomic task into semantic encoding; the posture calculation module is equivalent to the action space parameter calculation model in the aforementioned embodiment, which is used to predict the action space parameters of the robot; the action model is used to control the operation of the robot.

[0141] In this embodiment, the input parameters of the robot arm control model are the observed values ​​of the environment and the task instructions, wherein the observed values ​​of the environment are equivalent to the visual environment parameter samples in the aforementioned embodiment, and the task instructions are equivalent to the semantic coding in the aforementioned embodiment.

[0142] In the inference phase of the robot control model, the action parameters are continuously output to the robot until the task is completed or fails. In the training phase of the robot control model, after each action of the robot, the reward model is used to calculate the reward and update the model.

[0143] Specifically, the environmental perception module is used to calculate the depth information and semantic information of the robot arm's operating area environment through a trained artificial intelligence (AI) model, so that in the subsequent processing process, the depth information and semantic information can be used as feature inputs of the action model.

[0144] The observation values ​​of the environment in the work area can be obtained through monocular cameras, multi-cameras, depth cameras, laser radars, etc., and input into the environment perception module to obtain the depth information and semantic information of the environment. Furthermore, the 3D environment information can be constructed through the 3D reconstruction algorithm, and the visual feature R can be obtained through the pre-trained visual recognition algorithm. Among them, R is the output of the neural network before the last fully connected layer of the visual model.

[0145] The instruction semantics module is used to obtain context information of the instruction. The instruction semantics module can be a semantic macro model or a multimodal macro model. The instruction semantics module is used to decompose any task into atomic tasks in an atomic task list. Furthermore, a task instruction can be issued from the task list to the instruction semantics module, which is input into the instruction semantics module to obtain context information obtained by the instruction semantics module based on the task instruction.

[0146] For example, the instruction semantic model can be used to select the atomic task that is currently required to be performed and has not yet been completed from the atomic task list, and output its corresponding semantic code E.

[0147] The action determination module is used to determine whether the atomic task is completed through the trained or fine-tuned AI model. Specifically, the action determination model is used to obtain the execution result of the atomic task after the current atomic task is executed. The reward value is obtained according to the execution result. When the reward value is greater than the preset reward value threshold, the execution result of the atomic task is regarded as the successful execution of the task. Otherwise, the atomic task status is recorded as being executed, and the context information of the instruction semantic module is updated.

[0148] Specifically, the semantic encoding of the atomic task and the picture of the working area after the robot arm moves are input into the action judgment model to obtain the action execution result. For example, when the action is completed, the output execution result mark 1 is output, otherwise 0 is output.

[0149] In this embodiment, the robot control model constructed by reinforcement learning predicts the robot posture parameters at the current moment through the posture calculation module, and indirectly controls the action of the robot. The posture calculation module is used to obtain the predicted action space parameter information of the end of the robot based on the environment depth information D, image features R, and the atomic task semantic encoding output by the instruction semantic module.

[0150] The predicted action space parameter information includes the end position (x, y, z) of the robot arm, the end posture (Y, P, R), and the angle r of the robot arm's upper arm rotating around the shoulder-wrist connection line.

[0151] The observation input of the action model, the semantic input of the atomic instruction, the output posture parameters, and the reward value corresponding to the execution result after the robot arm runs according to the joint parameters corresponding to the predicted posture can be used as training data to iteratively optimize the model parameters of the robot arm control model until the model converges. For example, a random perturbation can be added to each variable output by the model during the model training stage.

[0152] Refer to the following Figure 5 and Figure 6 , the method for training the robot arm control model of the present application is further described:

[0153] In one embodiment, the robot arm control method may include the following steps:

[0154] Step S1, initialize the context of the atomic task in the instruction semantic module, initialize the environmental information, obtain the environmental information t at the current moment, and set the task to be executed.

[0155] The environmental information t in this step is Figure 6 The state t in .

[0156] Get the task instructions of the robot arm, initialize the context information of the task instructions, initialize the environment information of the working area, obtain the visual environment parameters at the current moment, and set the task that the robot arm is currently performing. Among them, the instruction semantic module can decompose the task instructions into a set of atomic tasks in the list through the semantic big model, and encode the atomic tasks into semantic features. Then, according to the current completion status of the atomic task, send the atomic task code to the action model.

[0157] Step S2, through the instruction semantic module, according to the task instruction and the context status information of the task instruction, select the atomic task that needs to be executed and is not completed from the atomic task list corresponding to the task to be executed currently, and output the semantic code E of the atomic task.

[0158] The context information of the task instruction may reflect the task executed by the previous task instruction of the current task instruction and the task executed by the next task instruction.

[0159] For example, a scenario task can be captured and an atomic task list can be designed, wherein the atomic task list includes multiple atomic tasks that need to be executed in sequence.

[0160] For example, the recorded action is segmented according to the predefined atomic tasks to obtain multiple segmented segments. Each segmented segment contains a set of sequences, wherein each element of each set of sequences contains: the state before the action, the current state includes the visual observation value of the robot camera, the joint angle before the action, the atomic task identifier, the executed action (for example, the executed joint rotation angle), and the state after the action is executed, wherein the state after the action is executed can be obtained through the visual observation value of the camera, and the state after the action is executed includes whether the atomic task is completed or not.

[0161] Step S3, obtaining depth information D and visual features R corresponding to the environmental information t through the environmental perception module according to the sensor used to obtain the environmental information t.

[0162] The environmental information t in this step is equivalent to Figure 6 The environmental features obtained by the environmental perception module.

[0163] For example, a monocular camera can be installed at the center of the gripper of the robot arm to obtain environmental information through the monocular camera. A red-green-blue depth (RGBD) camera can be installed at a fixed position in the environment without obvious blind spots to obtain environmental information through the RGBD camera to ensure that there are no large visual blind spots.

[0164] Step S4, obtaining the current joint parameters T of each joint of the robotic arm through the posture calculation module, and obtaining the posture parameters W of the robotic arm based on the forward and inverse kinematics equations.

[0165] Where W includes the end position (x, y, z), the end posture (Y, P, R), and the angle r of the upper arm rotating around the shoulder-wrist connection line. x, y, and z represent the coordinates of the X-axis, Y-axis, and Z-axis in the reference coordinate system, respectively, and Y, P, and R represent the yaw angle, pitch angle, and roll angle, respectively.

[0166] According to the 7 joint parameters of the robot arm, the end position (x, y, z), end posture (Y, P, R) of the robot arm, and the angle r of rotation of the robot arm around the shoulder-wrist connection line are calculated.

[0167] Step S5, obtaining semantic coding E, depth information D and visual features R, as well as the posture parameter W of the robotic arm through the action model, and obtaining the predicted posture parameter W1 of the robotic arm based on the posture prediction model.

[0168] Among them, W1 includes the coordinates of the end of the robotic arm (x1, y1, z1), the end posture (Y1, P1, R1), and the angle r1 of the upper arm rotating around the shoulder-wrist connection line.

[0169] x1, y1 and z1 represent the coordinates of the X-axis, Y-axis and Z-axis in the reference coordinate system, respectively; Y1, P1 and R1 represent the yaw angle, pitch angle and roll angle, respectively. Step S6, obtain the posture prediction parameter W1 through the posture calculation module, and calculate the joint parameter T1 of each joint of the robot arm through the forward and inverse kinematics equations with the range of motion of the elbow joint opening and closing as the constraint.

[0170] Specifically, the robot arm has 7 joints, of which the elbow joint is the 4th joint. The posture calculation module obtains the output W1 of the motion model and limits the preset range of motion of the 4th joint (elbow joint opening and closing) of the 7 joints to 0-180 degrees. Based on the preset range of motion, the predicted motion space parameters and the forward and inverse kinematics equations, the new 7 joint parameters T1 are calculated.

[0171] The posture calculation module is used to convert the posture parameters of the robot arm with the joint parameters through numerical calculation, thereby converting the action space of the action model from the original joint parameters to the posture parameters, promoting the convergence of the action model.

[0172] The processing process of the pose rotation joint parameters is as follows: limit the range of motion of joint 4 (elbow joint opening and closing) to 0-180 degrees. Input the end position (x, y, z) of the robot arm predicted by the motion model, the end posture (Y, P, R), the angle r of the robot arm's upper arm rotation around the shoulder-wrist connection line (see Figure 2 ).

[0173] Among them, according to the physical structure of the robot arm, the values ​​of each joint of the robot arm are calculated. If there is a solution, the unique solution is output, otherwise there is no solution.

[0174] Step S7, determine whether the forward and inverse kinematics equations have solutions, if so, proceed to step S8, otherwise proceed to step S12.

[0175] The forward and inverse kinematics equations have solutions, indicating that the robotic arm can move to the joint position corresponding to the predicted motion space parameters. The forward and inverse kinematics equations have no solutions, indicating that the robotic arm can move to the joint position corresponding to the predicted motion space parameters.

[0176] Step S8, controlling the movement of the robot arm according to the joint parameter T1, and updating the environmental information t to t1.

[0177] Adjust the joint parameters of the robot arm to t1, and obtain the updated environmental parameters according to the method of step S1.

[0178] Step S9, through the action determination module, obtains visual observation values ​​according to the sensor used to obtain environmental information t1, and determines whether the action is completed according to the action recognition model.

[0179] For example, based on the action determination module, visual observation values ​​are obtained from the sensor of the environmental information t1 to determine whether the action is completed.

[0180] The model training process is as follows: construct pictures of the atomic task completion status and the corresponding atomic task codes, construct a training set, train a small visual model or fine-tune a large visual model so that it can identify whether the atomic task is completed.

[0181] Step S10, if the action is completed, the observation values ​​Di, Ri, posture prediction Wi, and reward value Q(i) of each moment i of the atomic task from the beginning to the completion are transmitted to the action model in sequence.

[0182] Among them, Di, Ri and Wi are the depth information, visual features and posture at the i-th moment respectively, and Q(i) is the reward value at the i-th moment. The reward value is obtained by multiplying the maximum incentive value by the attenuation coefficient, where the maximum incentive value can be set according to user needs, for example, it can be set to 10000, 1000, or other values.

[0183] For example, a large model or a small model is visually recognized to determine whether an atomic task is completed, thereby determining a robotic arm reinforcement learning training method of a reward function.

[0184] Step S11, obtaining completion information of whether the atomic task is completed through the instruction semantic module, and updating the context of the atomic task according to the completion information.

[0185] When the atomic task is completed, the atomic task to be executed is updated to the next atomic task in the atomic task list.

[0186] Step S12, determine whether the execution times of the current atomic task reaches the set threshold value K, if not, return to step S2, if it reaches the set threshold value K, or the forward and inverse kinematics equations have no solution, enter step S13.

[0187] The threshold value K can be set according to user needs. For example, if the atomic task is to grasp an object from position A, and the threshold value is set to 20, the action of the robot arm from the current position to grasping the object A can be decomposed into 20 decomposed tasks. If the set threshold value K is reached, or the forward and inverse kinematics equations have no solution, it means that the task has failed.

[0188] Step S13, set the reward value to -Q(i), execute step S10, and then return to step S1.

[0189] Train the action model until the model converges on each atomic task. Use the action model to explore and learn new actions until the model converges.

[0190] In a new open scenario, a camera is installed at the same position of the robotic arm and at a fixed position in the environment without obvious blind spots. The trained robotic arm control model obtained in the embodiment can be used to plan and verify the robotic arm's actions.

[0191] In this embodiment, by pre-training the visual model, the tasks in the open multi-task scene can be decomposed, by training the visual model, the scene can be expanded, and by the 3D depth algorithm, the grasping and obstacle avoidance in the open scene can be realized; the model is connected in series by the reinforcement learning algorithm and the posture calculation module to realize the self-supervised training of the multi-task open scene grasping task without action annotation. In this embodiment, through task decomposition, deep information acquisition, and reward function design, self-supervised reinforcement learning training without action annotation can be realized, which reduces the cost of annotation compared with the methods of related technologies.

[0192] For example, the robotic arm is a robotic arm including 7 robotic arm joints. In the process of reinforcement learning, the action space of the action model is improved according to the physical characteristics of the 7-axis robotic arm. Specifically, the rotation angle of the upper arm around the shoulder-wrist joint line is increased on the basis of the end posture of the robotic arm. By limiting the opening and closing angle of joint 4 by the posture calculation module, it is possible to achieve a unique mapping of the action space to the joint angle of the robotic arm under the premise that the action space parameters are accessible, avoiding the situation where the joint parameters are mapped at the end posture of the 7-axis robotic arm, and the redundant robotic arm have infinite solutions, and avoiding the problem of nonlinear joint angle control in order to achieve the target posture, thereby improving the convergence of the model.

[0193] For example, the robot arm control method provided in the embodiment of the present application may include the following steps:

[0194] Step F1, initialize the context of the atomic task in the instruction semantic module, initialize the environmental information, obtain the environmental information t at the current moment, and set the task to be executed.

[0195] Get the task instructions of the robot arm, initialize the context information of the task instructions, initialize the environmental information of the working area, get the current environmental information, and set the task to be performed by the robot arm.

[0196] Step F2, through the instruction semantic module, according to the task instruction and the context status information of the task instruction, select the atomic task that needs to be executed and is not completed from the atomic task list corresponding to the task to be executed currently, and output the semantic code E of the atomic task.

[0197] The method of this step can refer to the description of step S2 and will not be repeated here.

[0198] Step F3, obtaining depth information D and visual features R corresponding to the environmental information t through the environmental perception module and according to the environmental information t obtained by the sensor.

[0199] The method of this step can refer to the description of the aforementioned step S3, which will not be repeated here.

[0200] Step F4, obtaining the current joint parameters T of each joint of the robotic arm through the posture calculation module, and obtaining the posture parameters W of the robotic arm based on the forward and inverse kinematics equations.

[0201] Among them, W includes the end position (x, y, z), the end posture (Y, P, R), and the angle r of the upper arm rotating around the shoulder-wrist connection line.

[0202] The method of this step can refer to the description of the aforementioned step S4, which will not be repeated here.

[0203] Step F5, obtain the semantic code E, depth information D and visual features R, as well as the posture parameter W of the robotic arm through the action model, and obtain the predicted action space parameter W1 of the robotic arm based on the posture prediction model.

[0204] Among them, the end position (x1, y1, z1), the end posture (Y1, P1, R1), and the angle r1 of the upper arm rotating around the shoulder-wrist connection line.

[0205] The method of this step can refer to the description of the aforementioned step S5, which will not be repeated here.

[0206] Step F6, obtain the position prediction action space parameter W1 through the posture calculation module, take the opening and closing range of the elbow joint as a constraint, and calculate the joint parameter T1 of each joint of the robotic arm through the forward and inverse kinematics equations.

[0207] The method of this step can refer to the description of the aforementioned step S6, which will not be repeated here.

[0208] Step F7, control the movement of the robot arm according to the joint parameter T1, and update the environmental information T to T1, and then return to step F2.

[0209] The method of this step can refer to the description of the aforementioned step S7, which will not be repeated here.

[0210] Reference Figure 7 The embodiment of the present application further provides a robot arm control model training device, wherein the robot arm includes at least one robot arm joint, and the device 40 includes:

[0211] The first acquisition module 401 is used to acquire the current atomic task sample of the robot arm, the current action space parameter sample, and the current visual environment parameter sample of the robot arm's operating area; the atomic task sample is obtained by decomposing the target task sample to be executed by the robot arm;

[0212] The second acquisition module 402 is used to acquire the predicted action space parameter sample of the manipulator according to the current atomic task sample, the current action space parameter sample, the current visual environment parameter sample, and the manipulator control model;

[0213] The third acquisition module 403 is used to control the operation of the robot arm according to the predicted action space parameter samples, and obtain updated visual environment parameter samples of the operation area of ​​the robot arm after the operation;

[0214] The fourth acquisition module 404 is used to obtain the execution result according to the updated visual environment information and the preset execution result judgment model, and train the robotic arm control model according to the execution result to obtain the trained robotic arm control model.

[0215] For example, the second acquisition module 402 may include:

[0216] The first acquisition submodule is used to input the atomic task sample into a preset semantic model to obtain a semantic coding sample of the atomic task sample; the semantic coding sample is used to reflect the semantic content of the atomic task sample;

[0217] The second acquisition submodule is used to obtain the predicted action space parameter samples of the robotic arm according to the semantic coding samples, the current action space parameter samples, the current visual environment parameters, and the robotic arm control model.

[0218] Optionally, the robotic arm includes seven robotic arm joints, and the third acquisition module 403 may include:

[0219] The third acquisition submodule is used to acquire the preset range of motion of the target joint in the robot arm joint;

[0220] The fourth acquisition submodule is used to obtain the joint parameter samples of the robot arm joints based on the predicted action space parameter samples and the preset joint parameter calculation model with the preset activity range as the constraint condition;

[0221] The control submodule is used to control the operation of the robot arm according to the joint parameter samples.

[0222] Optionally, the predicted motion space parameter sample includes at least one of the following: a predicted value of the posture of the end of the robotic arm, and a predicted value of the rotation angle of the robotic arm's upper arm around the shoulder-wrist connection line.

[0223] Optionally, the third acquisition module 403 includes:

[0224] The fifth acquisition submodule is used to obtain the current joint parameter samples of the robot arm joints;

[0225] The sixth acquisition submodule is used to input the current joint parameter sample into a preset motion space parameter calculation model to obtain the current motion space parameter sample; wherein the current motion space parameter sample includes the posture information of the end of the robotic arm and the rotation angle of the robotic arm around the shoulder-wrist connection line.

[0226] Optionally, the atomic task includes a plurality of decomposed actions executed sequentially, and the second acquisition module 402 includes:

[0227] The seventh acquisition submodule is used to obtain the current predicted action space parameter sample of the robot arm according to the current decomposed action, the current action space parameter sample, the current visual environment parameter sample, and the robot arm control model.

[0228] The third acquisition module 403 includes:

[0229] An eighth acquisition submodule is used to acquire the number of executed decomposition actions;

[0230] The first determination submodule is used to determine that the execution result is task completion when the number of executed actions is less than or equal to the total number of all decomposed actions and the robot arm has completed the current atomic task based on the updated visual environment information and the preset execution result judgment model.

[0231] The second determination submodule is used to determine that the execution result is that the task is not completed when the number of executed actions is greater than the total number of all decomposed actions, or the robot arm cannot reach the position corresponding to the current predicted action space parameter sample.

[0232] Optionally, the fourth acquisition module 404 includes:

[0233] The ninth acquisition submodule is used to obtain the predicted action space parameter samples, environmental information and reward values ​​corresponding to each decomposed action that has been executed when the execution result is task completion;

[0234] The tenth acquisition submodule is used to train the robotic arm control model according to the predicted action space parameter samples and reward values ​​corresponding to each decomposed action, so as to obtain a trained robotic arm control model.

[0235] Optionally, the tenth acquisition submodule includes:

[0236] A first acquisition unit is used to acquire the execution order of the decomposed actions in the multiple decomposed actions of the multiple atomic tasks;

[0237] A first determining unit, used to determine the attenuation coefficient of the decomposed action according to the execution order;

[0238] The second acquisition unit is used to acquire the product of the attenuation coefficient and the preset reward value, and determine the product as the reward value of the decomposition action.

[0239] In this embodiment, by predicting the predicted action space parameter samples of the robot arm and training the model according to the predicted action space parameter samples, compared with the joint angle prediction method in the related art, nonlinear joint angle prediction is avoided, which facilitates model convergence. By decomposing the target task samples, atomic task samples are obtained, and the robot arm control model is trained for each atomic task sample, the obtained robot arm control model has high accuracy. This embodiment does not need to obtain a large amount of manual operation data, and uses the manual operation data as a sample training model, thereby saving the step of obtaining training samples, reducing the amount of data processing, and improving the efficiency of model training.

[0240] Reference Figure 8 This embodiment further provides a robot arm control device, wherein the robot arm includes at least one robot arm joint, and the device 50 includes:

[0241] The fifth acquisition module 501 is used to acquire the current atomic task of the robot arm, the current posture information, and the current environmental information of the robot arm's operating area; the atomic task is a task obtained by decomposing the target task, and the target task sample is used to describe the task to be performed by the robot arm;

[0242] The sixth acquisition module 502 is used to obtain the predicted motion space parameters of the manipulator according to the atomic task, the current posture information, the current environment information, and the trained manipulator control model;

[0243] The seventh acquisition module 503 is used to control the operation of the robotic arm according to the predicted action space parameters; wherein the robotic arm control model is obtained according to the robotic arm control model training method of any one of the above embodiments.

[0244] In this embodiment, by predicting the predicted action space parameter samples of the robot arm, the model is trained according to the predicted action space parameter samples. Compared with the joint angle prediction method in the related art, nonlinear joint angle prediction is avoided, which facilitates model convergence. By decomposing the target task samples, atomic task samples are obtained, and the robot arm control model is trained for each atomic task sample, the obtained robot arm control model has high accuracy. Based on the robot arm control model, the posture parameters of the robot arm are predicted, and based on the posture parameters, the operation of the robot arm can be accurately controlled.

[0245] The present invention also provides an electronic device, see Fig. 9, including: a processor 601, a memory 602, and a computer program 6021 stored in the memory and executable on the processor, and when the processor executes the program, the robot arm control model training method or the robot arm control method of the aforementioned embodiment is implemented.

[0246] The present invention also provides a readable storage medium. When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the robotic arm control model training method or the robotic arm control method of the aforementioned embodiment.

[0247] The present invention also provides a robotic arm, comprising the robotic arm control model training device, the robotic arm control device, or the electronic device of the aforementioned embodiment.

[0248] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0249] It should be noted that the various information and data obtained in the embodiments of the present invention are all obtained with the authorization of the information / data holder. The algorithms and displays provided herein are not inherently related to any specific computer, virtual system or other device. Various general systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language. It should be understood that the content of the present invention described herein can be implemented using various programming languages, and the description of specific languages ​​above is to disclose the best mode of implementation of the present invention.

[0250] In the specification provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this specification. Similarly, it should be understood that in order to simplify the present invention and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: the claimed invention requires more features than the features clearly recorded in each claim. More specifically, as reflected in the following claims, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby explicitly incorporated into the specific embodiment, wherein each claim itself is a separate embodiment of the present invention.

[0251] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0252] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in the sorting device according to the present invention. The present invention may also be implemented as a device or apparatus program for executing part or all of the methods described herein. Such a program for implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0253] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0254] The user information (including but not limited to the user's device information, user personal information, etc.) and related data involved in the present invention are all information authorized by the user or by all parties.

[0255] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention. The above are only specific implementation methods of the present invention, but the protection scope of the present invention is not limited to this. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for training a robot arm control model, characterized in that: include: Obtain the current atomic task sample of the robot arm, the current action space parameter sample, and the current visual environment parameter sample of the robot arm's operating area; The atomic task sample is obtained by decomposing the target task sample to be performed by the robot arm; Acquire a predicted action space parameter sample of the robotic arm according to the current atomic task sample, the current action space parameter sample, the current visual environment parameter sample, and the robotic arm control model; Controlling the operation of the robot arm according to the predicted action space parameter samples, and obtaining updated visual environment parameter samples of the operating area of ​​the robot arm after the operation; The execution result is obtained according to the updated visual environment information and the preset execution result judgment model, and the robotic arm control model is trained according to the execution result to obtain a trained robotic arm control model.

2. The method according to claim 1, characterized in that The step of obtaining the predicted action space parameter sample of the manipulator according to the current atomic task sample, the current action space parameter sample, the current visual environment parameter sample, and the manipulator control model includes: Inputting the atomic task sample into a preset semantic model to obtain a semantic coding sample of the atomic task sample; the semantic coding sample is used to reflect the semantic content of the atomic task sample; According to the semantic coding samples, the current action space parameter samples, the current visual environment parameters, and the robotic arm control model, a predicted action space parameter sample of the robotic arm is obtained.

3. The method according to claim 1, characterized in that The robotic arm includes seven robotic arm joints, and controlling the operation of the robotic arm according to the predicted motion space parameter sample includes: Get the preset range of motion of the target joint in the robot arm joint; Taking the preset range of motion as a constraint condition, obtaining joint parameter samples of the mechanical arm joints according to the predicted motion space parameter samples and a preset joint parameter calculation model; The operation of the robotic arm is controlled according to the joint parameter samples.

4. The method according to claim 1, characterized in that: The predicted motion space parameter samples include at least one of the following: a predicted value of the posture of the end of the robotic arm, and a predicted value of the rotation angle of the robotic arm around the shoulder-wrist connection line.

5. The method according to claim 1, characterized in that Get the current motion space parameter sample of the robot arm, including: Get the current joint parameter samples of the robot arm joints; Inputting the current joint parameter sample into a preset motion space parameter calculation model to obtain the current motion space parameter sample; The current motion space parameter sample includes the posture information of the end of the robotic arm and the rotation angle of the robotic arm around the shoulder-wrist connection line.

6. The method according to claim 1, characterized in that The atomic task includes a plurality of decomposed actions that are executed in sequence. According to the current atomic task sample, the current action space parameter sample, the current visual environment parameter sample, and the robot control model, a predicted action space parameter sample of the robot is obtained, including: According to the current decomposed action, the current action space parameter sample, the current visual environment parameter sample, and the robot control model, the current predicted action space parameter sample of the robot is obtained.

7. The method according to claim 6, characterized in that The execution result is obtained according to the updated visual environment information and the preset execution result judgment model, including: Get the number of executed decomposition actions; When the number of executed actions is less than or equal to the total number of all decomposed actions, and when it is determined that the robot arm has completed the current atomic task based on the updated visual environment information and the preset execution result judgment model, the execution result is determined to be task completion; When the number of executed actions is greater than the total number of all decomposed actions, or the robot arm cannot reach the position corresponding to the current predicted action space parameter sample, it is determined that the execution result is that the task is not completed.

8. The method according to claim 6, characterized in that Training the robot arm control model according to the execution result to obtain a trained robot arm control model includes: When the execution result is task completion, obtaining predicted action space parameter samples, environmental information, and reward values ​​corresponding to each executed decomposed action; The robotic arm control model is trained according to the predicted action space parameter samples and reward values ​​corresponding to each decomposed action to obtain a trained robotic arm control model.

9. The method according to claim 8, characterized in that Get the reward value of the decomposed action, including: Obtaining the execution order of decomposed actions in multiple decomposed actions of the atomic task; Determining the attenuation coefficient of the decomposed action according to the execution order; The product of the attenuation coefficient and a preset reward value is obtained, and the product is determined as the reward value of the decomposition action.

10. A method for controlling a robotic arm, characterized in that: include: Obtain the current atomic task of the robot, the current action space parameters, and the current visual environment parameters of the robot's operating area; The atomic task is obtained by decomposing the target task to be performed by the robot arm; Obtaining predicted motion space parameters of the robotic arm according to the atomic task, the current motion space parameters, the current visual environment parameters, and the trained robotic arm control model; The operation of the robotic arm is controlled according to the predicted motion space parameters; the robotic arm control model is obtained according to the robotic arm control model training method according to any one of claims 1 to 9.

11. A robot arm control model training device, characterized in that: The device comprises: The first acquisition module is used to acquire the current atomic task sample of the robot arm, the current action space parameter sample, and the current visual environment parameter sample of the robot arm's operating area; the atomic task sample is obtained by decomposing the target task sample to be executed by the robot arm; A second acquisition module is used to acquire a predicted action space parameter sample of the manipulator according to the current atomic task sample, the current action space parameter sample, the current visual environment parameter sample, and the manipulator control model; A third acquisition module is used to control the operation of the robot arm according to the predicted action space parameter samples, and obtain updated visual environment parameter samples of the operating area of ​​the robot arm after the operation; The fourth acquisition module is used to obtain the execution result according to the updated visual environment information and the preset execution result judgment model, and train the robotic arm control model according to the execution result to obtain the trained robotic arm control model.

12. A robot arm control device, characterized in that: The device comprises: A fifth acquisition module is used to acquire the current atomic task of the robot arm, the current action space parameters, and the current visual environment parameters of the robot arm's operating area; the atomic task is obtained by decomposing the target task to be performed by the robot arm; a sixth acquisition module, configured to obtain predicted motion space parameters of the robotic arm according to the atomic task, the current motion space parameters, the current visual environment parameters, and the trained robotic arm control model; A seventh acquisition module is used to control the operation of the robotic arm according to the predicted action space parameters; the robotic arm control model is obtained according to the robotic arm control model training method according to any one of claims 1 to 9.

13. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 10 is implemented.

14. A readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method as claimed in any one of claims 1 to 10.

15. A robotic arm, characterized in that: The robotic arm includes the robotic arm control model training device as claimed in claim 11, the robotic arm control device as claimed in claim 12, or the electronic device as claimed in claim 13.

Images