Mechanical arm control method and device based on meta-reinforcement learning, equipment and medium

Abstract

The embodiment of the application provides a kind of based on meta reinforcement learning's mechanical arm control method and device, equipment and medium, belong to mechanical arm control technical field.The method comprises: obtaining the historical interaction data of mechanical arm control task;Based on historical interaction data, the feature extraction is carried out to mechanical arm control task, and task feature is obtained;Based on the task feature, determine target mechanical arm action by preset policy network;The advantage value of the target mechanical arm action is calculated by preset advantage function, and target action advantage is obtained;Target action advantage is normalized, and target action weight is obtained;Based on target action weight, the parameter updating is carried out to preset policy network, and target policy network is obtained;Mechanical arm control is carried out using target policy network.The embodiment of the application can improve the generalization ability of mechanical arm control method.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mechanical arm control, and particularly relates to a mechanical arm control method and device based on meta-reinforcement learning, equipment and a medium. BACKGROUND

[0002] As the core actuator of industrial automation, medical surgery and service robots, the motion control of the mechanical arm needs to meet the real-time response requirements of high precision, high adaptability and dynamic environment. However, the current traditional control methods (such as PID control, adaptive control and the like) depend on accurate mathematical models, but the generalization ability of such mathematical models is low when facing complex dynamic tasks, and it is difficult to adapt to the diversity of tasks. Therefore, how to improve the generalization ability of the mechanical arm control method has become a technical problem to be solved. SUMMARY

[0003] The main purpose of the embodiments of the present application is to provide a mechanical arm control method and device based on meta-reinforcement learning, equipment and a medium, which aims to improve the generalization ability of the mechanical arm control method.

[0004] To achieve the above purpose, a first aspect of the embodiments of the present application provides a mechanical arm control method based on meta-reinforcement learning, which comprises:

[0005] Obtaining historical interaction data of a mechanical arm control task;

[0006] Based on the historical interaction data, performing feature extraction on the mechanical arm control task to obtain task features;

[0007] Based on the task features, determining a target mechanical arm action through a preset policy network;

[0008] Calculating the advantage value of the target mechanical arm action through a preset advantage function to obtain a target action advantage;

[0009] Performing normalization processing on the target action advantage to obtain a target action weight;

[0010] Based on the target action weight, updating the parameters of the preset policy network to obtain a target policy network;

[0011] Using the target policy network to control the mechanical arm.

[0012] To achieve the above purpose, a second aspect of the embodiments of the present application provides a mechanical arm control device based on meta-reinforcement learning, which comprises:

[0013] An interaction data acquisition module is configured to obtain historical interaction data of a mechanical arm control task;

[0014] The task feature extraction module is configured to perform feature extraction on the robot control task based on historical interaction data to obtain task features.

[0015] The action and state determination module is configured to determine target robot actions based on the task features through a preset policy network.

[0016] The target action advantage calculation module is configured to calculate the advantages of the target robot actions through a preset advantage function to obtain target action advantages.

[0017] The normalization processing module is configured to normalize the target action advantages to obtain target action weights.

[0018] The parameter updating module is configured to update the parameters of the preset policy network based on the target action weights to obtain a target policy network.

[0019] The robot control module is configured to control the robot using the target policy network.

[0020] To achieve the above object, a third aspect of the embodiments of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method of the first aspect when executing the computer program.

[0021] To achieve the above object, a fourth aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method of the first aspect.

[0022] The robot control method and device based on meta-reinforcement learning, the equipment and the medium provided by the present application can generate task features representing task characteristics by extracting features from historical interaction data; the strategy network can select actions guided by the task features to improve the adaptability of the strategy network to different tasks; the target action advantage can be calculated by a preset advantage function to quantify the advantages and disadvantages of the target robot actions; the target action advantage is normalized to convert it into a weight, and the parameters of the preset strategy network are updated based on the target action weight to ensure that high-advantage actions obtain high weights to dominate the training direction, so that the strategy network can quickly adapt to unknown tasks with only a small amount of samples, and the generalization ability of the robot control method is improved. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is a flowchart of the robot control method based on meta-reinforcement learning provided by the embodiments of the present application;

[0024] Figure 2 is a flowchart of meta-reinforcement learning;

[0025] Figure 3Figure 3 is another flowchart of the method for controlling a robot arm based on meta reinforcement learning provided by an embodiment of the present application;

[0026] Figure 4 is a structural schematic diagram of the device for controlling a robot arm based on meta reinforcement learning provided by an embodiment of the present application;

[0027] Figure 5 is a hardware structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0029] It should be noted that although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0031] First, the meanings of several terms involved in the present application are analyzed:

[0032] Meta reinforcement learning (Meta-RL): Meta reinforcement learning is a machine learning method that aims to train an agent (such as an AI agent) to quickly adapt to new tasks by learning a general learning strategy across multiple tasks, so as to make efficient decisions in unseen environments. Meta reinforcement learning belongs to a subfield of artificial intelligence, which is the intersection of reinforcement learning and meta learning, and is also commonly referred to as meta reinforcement learning or Meta-RL. Meta reinforcement learning includes meta policy optimization, model-agnostic meta learning (MAML) and gradient-based adaptation algorithms. It is commonly used in technical fields such as robot control, game agent development, autonomous driving systems, resource management and personalized recommendation.

[0033] Meta training: Meta training is the training phase in the meta-learning framework, which involves optimizing a meta-model (such as a neural network) on multiple related tasks to learn shared knowledge across tasks, enabling the model to generalize to new tasks. Meta training is part of the machine learning process and is the core step of meta-learning algorithm design and implementation, often referred to as the meta-optimization phase. Meta training includes task sampling, meta-loss function design, inner loop training, and outer loop updating.

[0034] Meta testing: Meta testing is the evaluation phase in the meta-learning framework, which is used to test the generalization ability and adaptation efficiency of the meta-trained model on new tasks, and to verify the performance of the model by simulating unknown environments. Meta testing is part of the machine learning process and is a key step in meta-learning verification and deployment, often referred to as the meta-evaluation phase. Meta testing includes task adaptability testing, performance measurement (such as accuracy or reward score), and robustness analysis.

[0035] In related technologies, Figure 2 In offline meta-reinforcement learning, because the tasks used in the meta-training phase and the meta-testing phase are different, the tasks used in the meta-training phase are constructed by a task distribution based on historical data or a predefined task library, while the meta-testing phase handles completely new tasks. The tasks in the meta-testing phase do not appear in the meta-training tasks, resulting in a distribution shift problem. The distribution shift problem can cause insufficient generalization performance of the agent, making the agent unable to effectively fine-tune with a small number of samples and poor generalization ability.

[0036] Therefore, the embodiment of the present application provides a meta-reinforcement learning-based mechanical arm control method and device, equipment and medium, which aims to improve the generalization ability of the mechanical arm control method.

[0037] The meta-reinforcement learning-based mechanical arm control method and device, electronic equipment and storage medium provided by the embodiment of the present application are specifically explained by the following embodiments. First, the meta-reinforcement learning-based mechanical arm control method in the embodiment of the present application is described.

[0038] The embodiment of the present application can acquire and process related data based on artificial intelligence technology. Artificial intelligence (AI) is the use of digital computers or computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0039] The artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric identification technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0040] The method for controlling a mechanical arm based on meta-reinforcement learning provided by the embodiments of the present application relates to the technical field of mechanical arm control. The method for controlling a mechanical arm based on meta-reinforcement learning provided by the embodiments of the present application can be applied in a terminal, can also be applied in a server end, and can also be software running in the terminal or the server end. In some embodiments, the terminal can be a notebook computer, a desktop computer, etc.; the server end can be configured as an independent physical server, can also be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform; and the software can be an application for implementing the method for controlling a mechanical arm based on meta-reinforcement learning, etc., but is not limited to the above forms.

[0041] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, multiprocessor systems, microprocessor-based systems, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0042] It should be noted that in various specific embodiments of the present application, when relevant processing needs to be performed on data related to the identity or characteristics of the user, such as user information, user behavior data, user history data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the user's separate permission or separate consent will be obtained through a pop-up window or by jumping to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for the normal operation of the embodiments of the present application will be obtained.

[0043] Figure 1 is an optional flowchart of the mechanical arm control method based on meta-reinforcement learning provided by the embodiments of the present application, Figure 1 The method in can include but is not limited to steps S101-S107.

[0044] Step S101, obtaining historical interaction data of a mechanical arm control task;

[0045] Step S102, based on the historical interaction data, performing feature extraction on the mechanical arm control task to obtain task features;

[0046] Step S103, based on the task features, determining a target mechanical arm action through a preset policy network;

[0047] Step S104, calculating an advantage value of the target mechanical arm action through a preset advantage function to obtain a target action advantage;

[0048] Step S105, normalizing the target action advantage to obtain a target action weight;

[0049] Step S106, based on the target action weight, updating parameters of the preset policy network to obtain a target policy network;

[0050] Step S107, using the target policy network for mechanical arm control.

[0051] As can be easily understood, the historical interaction data includes trajectory data generated by the interaction between the agent and the environment in a multi-task scenario, such as action-state-reward sequences. The target mechanical arm action can include the action probability distribution output by the policy network (i.e., the probability value of selecting different actions), and the preset advantage function is used to quantify the advantages and disadvantages of each action. The target action advantage is normalized to obtain the target action weight. Based on the target action weight, the parameters of the preset policy network are updated to obtain the target policy network. In the meta-test phase, the environment state and the task features of the new task are input to the target policy network, and the next mechanical arm action of the target policy network is used to complete tasks such as grasping and stacking.

[0052] The steps S101 to S107 shown in the embodiments of the present application can generate task features representing task characteristics by feature extraction on historical interaction data; can improve the adaptability of the policy network to different tasks by guiding the policy network to select actions through the task features; can quantify the advantages and disadvantages of the target robot action by calculating the target action advantage through the preset advantage function; can normalize the target action advantage to convert it into a weight, and update the parameters of the preset policy network based on the target action weight to ensure that high-advantage actions obtain high weights to dominate the training direction, so that the policy network can quickly adapt to unknown tasks with only a small amount of samples, thereby improving the generalization ability of the robot control method.

[0053] The embodiments of the present application can be applied to the motion control of various robots and robots, such as industrial robots deployed on a production line, robots required to perform complex tasks, and actual application scenarios such as performing related work in a space station. Taking a robot performing a complex task as an example, the robot needs to perform a variety of tasks with complex procedures and high automation, such as picking up certain objects, opening a door lock, etc. By applying the method described in the embodiments of the present application, interaction data between the robot and the environment of a certain type and number of tasks is collected for training, and the target policy network robot can well complete similar tasks, effectively making up for the shortcomings of traditional methods and reducing the use cost of the robot.

[0054] In step S102 of some embodiments, the interaction data can be encoded to conform to a Gaussian posterior distribution by a preset inference network, and the posterior distribution parameters z (such as mean and variance) of the robot control task are obtained as task features.

[0055] In some embodiments, the robot control method based on meta-reinforcement learning further includes: setting the loss function of the inference network as wherein β represents a regularization coefficient, represents the inference network, which is used to output the posterior distribution parameter z based on the input data (i.e., the interaction data c), and p(z) = N(0, I). The embodiments of the present application limit the information amount of the latent posterior distribution parameter z through the KL divergence, which essentially limits the inference network the difference between the output data and the prior distribution p(z) (usually assumed to be a normal distribution N(0, I)), if If the output data carries too much task-unrelated information, the KL value will increase. By minimizing the KL divergence, the distribution of z is forced to approach a simple prior distribution, thereby limiting the information amount of z and preventing overfitting of the extracted latent posterior distribution parameter z due to insufficient data.

[0056] In some embodiments, step S103 can include but is not limited to steps S301 to S302:

[0057] In step S301, an initial environment state is obtained.

[0058] In step S302, an action is selected by a preset policy network according to a task feature and the initial environment state, to obtain a target robot arm action.

[0059] As can be easily understood, the initial environment state can include a robot arm pose, a robot arm position, an object position, etc. The target robot arm action can be a motion parameter of each joint of the robot arm. After the target robot arm action is executed, the environment state can be updated based on environment feedback to obtain a target environment state.

[0060] In some embodiments, step S104 can include but is not limited to steps S401 to S403:

[0061] In step S401, a value of the target robot arm action is calculated by a preset action value network, to obtain a target action value.

[0062] In step S402, a value of the initial environment state is calculated by a preset state value network, to obtain an initial state value.

[0063] In step S403, an advantage value of the target robot arm action is calculated by a preset advantage function according to the target action value and the initial state value, to obtain a target action advantage.

[0064] In step S403 of some embodiments, a difference between the target action value and the initial state value is taken as the target action advantage. Specifically, the advantage function is set as A(s, a) = Q(s, a) - V(s), wherein Q(s, a) is the target action value output by an action value network (i.e., a Q value network), and V(s) is the initial state value output by a state value network (i.e., a V value network).

[0065] In some embodiments, the robot arm control method based on meta-reinforcement learning further includes steps S501 to S506:

[0066] In step S501, an environment state is updated according to the target robot arm action and the initial environment state, to obtain a target environment state.

[0067] In step S502, an average value of a task feature is calculated, to obtain a task feature mean value.

[0068] In step S503, a value of the target environment state is calculated based on the task feature mean value, to obtain a target state value.

[0069] In step S504, a reward of the target robot arm action is obtained.

[0070] Step S505, based on the target action value, the reward and the target state value, an action value loss is calculated;

[0071] Step S506, based on the action value loss, the parameter of the action value network is updated.

[0072] In some embodiments, the meta-reinforcement learning based robot arm control method further comprises steps S601 to S602:

[0073] Step S601, based on the target action value and the initial state value, a state value loss is calculated;

[0074] Step S602, based on the state value loss, the parameter of the state value network is updated.

[0075] Specifically, referring to Figure 3 , trajectory data generated by the interaction between the agent and the environment when the agent performs different tasks, such as Markov decision tuples (action-state-reward), etc. are collected. Then, the Markov decision process is modeled by using the probability modeling method to obtain the task distribution p(T), T={S, A, T, R}, wherein S is the state space, A is the action space, T is the state transition probability, and R is the reward function. The trajectory data T i of the i-th task is randomly sampled from the task distribution p(T), and then stored in the experience replay buffer as historical interaction data. The historical interaction data is extracted by the Transformer model to generate task features z. According to the task features z and the current policy, the action probability distribution is output by the policy network (Actor). The action value network (Critic) includes two Q value networks (Q1 and Q2), which output the action value (Q value) according to the task features z and the action probability distribution. The state value network (Value Network) outputs the state value (V value) according to the state.

[0076] The V value loss function is set to calculate the state value loss In the formula, V(s, z) is the initial state value; Q θ (s, a, z) is the target action value; E (s,a,r,s′) represents the expectation of the joint distribution of the initial state, action, reward and target state.

[0077] The Q value loss function is set to calculate the action value loss In the formula, represents the expectation of the joint distribution of the initial state, action, reward and target state sampled from the experience replay buffer; Q θ(s, a, z) is a target action value; r is a reward; is a target state value, represents the average value of the sampled characteristics of multiple tasks; s and s' represent the current state (i.e., the initial environment state) and the next state (i.e., the target environment state), respectively; a is the current action (i.e., the target robot arm action). In the training process, the Q θ gradually approaches the optimal value function.

[0078] The embodiments of the present application update the V-value network parameters through the environment state s and the latent posterior distribution parameter z, and update the Q-value network parameters through the V-value network output, thereby ensuring the conservatism of the Q-value network estimation, so as to avoid performance degradation and error due to overestimation of the action value.

[0079] In step S105 of some embodiments, the target action advantage can be normalized by a softmax to obtain a target action weight ω(s, a).

[0080] In step S106 of some embodiments, the target action weight can be multiplied by the action probability distribution output by the policy network to obtain a weighted action probability distribution; based on the weighted action probability distribution, a policy loss weighted value is calculated, and the policy loss weighted value is minimized by gradient descent to obtain a target policy network.

[0081] In some embodiments, step S106 can include but is not limited to steps S701 to S705:

[0082] Step S701, according to the target action weight, the weighted value of the probability distribution of the target robot arm action is calculated to obtain a first distribution;

[0083] Step S702, the action value of the target robot arm action is calculated to obtain a target action value;

[0084] Step S703, the probability distribution of the target action value is calculated to obtain a second distribution;

[0085] Step S704, the difference between the first distribution and the second distribution is calculated to obtain a policy loss;

[0086] Step S705, based on the policy loss, the parameters of the policy network are updated to obtain a target policy network.

[0087] It is easy to understand that the first distribution representation strategy network represents the probability distribution of different actions taken by the strategy network under the current policy parameters for a specific state and task feature. The second distribution represents the probability distribution of different action values given the state and task feature, representing the optimal distribution of action values. The policy loss measures the performance of the policy network by calculating the difference between the first distribution and the second distribution, and updates the parameters of the policy network by minimizing the policy loss, so that the policy network can quickly adapt and fine-tune when facing new tasks, thereby improving the operation performance and efficiency of the robot arm in complex environments.

[0088] The steps S701 to S705 shown in the embodiments of the present application perform weighting processing on the probability distribution of the target robot arm action based on the target action weight to generate a first distribution, giving higher weights to high-advantage actions to highlight their contribution; calculate the action value of the target robot arm action to obtain the target action value, and quantify the expected cumulative return of the action; the target action value probability distribution is used as a second distribution; generate a policy loss by calculating the difference between the first distribution and the second distribution, the policy loss reflects the deviation of the current policy network output from the value optimal distribution, can convert the action advantage information into a directional optimization signal of the policy network, forcing the policy network parameters to update in the direction of increasing the probability of high-value actions, thereby achieving more accurate action selection strategy in robot arm control tasks, accelerating the convergence of the policy network and improving the sample utilization efficiency.

[0089] In step S704 of some embodiments, the policy loss is set as:

[0090]

[0091] wherein, is the policy loss; is used to ensure that the product of the target action weight ω(s, a) and the policy network π(a|s) forms a valid probability distribution in the action space.

[0092] In step S705 of some embodiments, based on the policy loss gradient descent is used to update the parameters of the policy network. After the meta-training phase is completed, the trained policy network can be used in the meta-testing phase to fine-tune for new tasks that have not been seen before.

[0093] Please refer to Figure 4 The embodiments of the present application also provide a robot arm control device based on meta-reinforcement learning, which can implement the robot arm control method based on meta-reinforcement learning described above, and the device comprises:

[0094] An interaction data acquisition module is configured to acquire historical interaction data of the robot arm control task.

[0095] The task feature extraction module is used to extract features from the robotic arm control task based on historical interaction data to obtain task features;

[0096] The action and state determination module is used to determine the target robotic arm action based on the task characteristics through a preset strategy network;

[0097] The target action advantage calculation module is used to calculate the advantage value of the target robotic arm action through a preset advantage function to obtain the target action advantage;

[0098] The normalization module is used to normalize the target action advantage to obtain the target action weight;

[0099] The parameter update module is used to update the parameters of the preset policy network based on the target action weights to obtain the target policy network.

[0100] The robotic arm control module is used to control the robotic arm using a target policy network.

[0101] The specific implementation of the meta-reinforcement learning-based robotic arm control device is basically the same as the specific implementation of the meta-reinforcement learning-based robotic arm control method described above, and will not be repeated here.

[0102] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described robotic arm control method based on meta-reinforcement learning. This electronic device can be any intelligent terminal such as an in-vehicle computer.

[0103] Please see Figure 5 , Figure 5 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0104] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), GPU (Graphics Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0105] The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 to implement the meta-reinforcement learning-based robotic arm control method of the embodiments of this application.

[0106] The input / output interface 903 is used to implement information input and output;

[0107] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0108] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0109] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0110] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described robotic arm control method based on meta-reinforcement learning.

[0111] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0112] The robotic arm control method, device, electronic device, and storage medium based on meta-reinforcement learning provided in this application can generate task features that characterize task properties by extracting features from historical interaction data. These task features guide the policy network in selecting actions, improving the network's adaptability to different tasks. A preset advantage function is used to calculate the advantage of the target action, quantifying the quality of the target robotic arm action. The advantage of the target action is normalized and converted into weights. Based on these weights, the parameters of the preset policy network are updated, ensuring that high-advantage actions receive high weights to dominate the training direction. Thus, the policy network can quickly adapt to unknown tasks with only a small number of samples, improving the generalization ability of the robotic arm control method.

[0113] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0114] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0115] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0116] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0117] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0118] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0119] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0120] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0121] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0122] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0123] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A robotic arm control method based on meta-reinforcement learning, characterized in that, The method includes: Acquire historical interaction data of robotic arm control tasks; Based on the historical interaction data, feature extraction is performed on the robotic arm control task to obtain task features; Based on the task characteristics, the target robotic arm action is determined through a preset strategy network; The target action value is obtained by calculating the value of the target robotic arm action through a preset action value network; The value of the initial environment state is calculated by using a pre-defined state value network, thus obtaining the initial state value. Based on the target action value and the initial state value, the advantage value of the target robotic arm action is calculated using a preset advantage function to obtain the target action advantage; Based on the target robotic arm's movements and the initial environmental state, the environmental state is updated to obtain the target environmental state; Calculate the average value of the task features to obtain the mean value of the task features; Based on the average of the task characteristics, the value of the target environment state is calculated to obtain the target state value; Obtain a reward for the target robotic arm's action; Based on the target action value, the reward, and the target state value, the action value loss is calculated. Based on the action value loss, the parameters of the action value network are updated; The target action advantage is normalized to obtain the target action weight; Based on the target action weight, the weighted value of the probability distribution of the target robotic arm action is calculated to obtain the first distribution; Calculate the motion value of the target robotic arm's action to obtain the target motion value; Calculate the probability distribution of the value of the target action to obtain the second distribution; Calculate the difference between the first distribution and the second distribution to obtain the policy loss; Based on the policy loss, the parameters of the preset policy network are updated to obtain the target policy network; The robotic arm is controlled using the target policy network.

2. The method according to claim 1, characterized in that, The step of determining the target robotic arm action based on the task characteristics through a preset policy network includes: Obtain the initial environment state; Based on the task characteristics and the initial environmental state, the target robotic arm action is obtained by selecting an action through a preset strategy network.

3. The method according to claim 1, characterized in that, The method further includes: Based on the target action value and the initial state value, the state value loss is calculated; Based on the state value loss, the parameters of the state value network are updated.

4. The method according to claim 1, characterized in that, Based on the historical interaction data, feature extraction is performed on the robotic arm control task to obtain task features, including: The historical interaction data is encoded using a pre-defined inference network to obtain the posterior distribution parameters of the robotic arm control task as the task features.

5. A robotic arm control device based on meta-reinforcement learning, characterized in that, The device includes: The interactive data acquisition module is used to acquire historical interactive data of the robotic arm control tasks; The task feature extraction module is used to extract features from the robotic arm control task based on the historical interaction data to obtain task features; The action and state determination module is used to determine the target robotic arm action based on the task characteristics through a preset strategy network; The target action advantage calculation module is used to calculate the value of the target robotic arm action through a preset action value network to obtain the target action value; calculate the value of the initial environment state through a preset state value network to obtain the initial state value; and calculate the advantage value of the target robotic arm action based on the target action value and the initial state value through a preset advantage function to obtain the target action advantage. The device is further configured to: update the environment state based on the target robotic arm action and the initial environment state to obtain the target environment state; calculate the average value of the task features to obtain the task feature mean; calculate the value of the target environment state based on the task feature mean to obtain the target state value; obtain the reward for the target robotic arm action; calculate the action value loss based on the target action value, the reward, and the target state value; and update the parameters of the action value network based on the action value loss. The normalization processing module is used to normalize the target action advantage to obtain the target action weight; The parameter update module is used to calculate the weighted value of the probability distribution of the target robotic arm action based on the target action weight to obtain a first distribution; calculate the action value of the target robotic arm action to obtain the target action value; calculate the probability distribution of the target action value to obtain a second distribution; calculate the difference between the first distribution and the second distribution to obtain the policy loss; and update the parameters of the preset policy network based on the policy loss to obtain the target policy network. A robotic arm control module is used to control the robotic arm using the target policy network.

6. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the robotic arm control method based on meta-reinforcement learning as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the robotic arm control method based on meta-reinforcement learning as described in any one of claims 1 to 4.

Images