Mechanical arm control method and device based on element reinforcement learning, equipment and medium

Abstract

The embodiment of the invention provides a mechanical arm control method and device based on meta reinforcement learning, equipment and a medium, and belongs to the technical field of mechanical arm control. The method comprises the following steps: acquiring historical interaction data of a mechanical arm control task; based on the historical interaction data, feature extraction is conducted on the mechanical arm control task, and task features are obtained; based on the task features, target mechanical arm actions are determined through a preset strategy network; calculating an advantage value of the target mechanical arm action through a preset advantage function to obtain a target action advantage; performing normalization processing on the target action advantages to obtain a target action weight; based on the target action weight, performing parameter updating on a preset policy network to obtain a target policy network; and mechanical arm control is conducted through the target strategy network. According to the embodiment of the invention, the generalization ability of the mechanical arm control method can be improved.

Description

Technical Field

[0001] This application relates to the field of robotic arm control technology, and in particular to a robotic arm control method, device, equipment and medium based on meta-reinforcement learning. Background Technology

[0002] As the core actuator in industrial automation, medical surgery, and service robots, robotic arms require motion control that meets the demands of high precision, high adaptability, and real-time response in dynamic environments. However, current traditional control methods (such as PID control and adaptive control) rely on precise mathematical models, but these models have low generalization ability when faced with complex dynamic tasks and are difficult to adapt to the diversity of tasks. Therefore, improving the generalization ability of robotic arm control methods has become an urgent technical problem to be solved. Summary of the Invention

[0003] The main objective of this application is to propose a robotic arm control method, device, equipment, and medium based on meta-reinforcement learning, aiming to improve the generalization ability of the robotic arm control method.

[0004] To achieve the above objectives, a first aspect of this application proposes a robotic arm control method based on meta-reinforcement learning, the method comprising:

[0005] Acquire historical interaction data of robotic arm control tasks;

[0006] Based on historical interaction data, feature extraction is performed on the robotic arm control task to obtain task features;

[0007] Based on the task characteristics, the target robotic arm action is determined through a preset strategy network;

[0008] The advantage of the target action is obtained by calculating the advantage value of the target robotic arm action through a preset advantage function;

[0009] The target action advantage is normalized to obtain the target action weight;

[0010] Based on the target action weights, the parameters of the preset policy network are updated to obtain the target policy network;

[0011] Using a target policy network for robotic arm control.

[0012] To achieve the above objectives, a second aspect of this application proposes a robotic arm control device based on meta-reinforcement learning, the device comprising:

[0013] The interactive data acquisition module is used to acquire historical interactive data of the robotic arm control tasks;

[0014] The task feature extraction module is used to extract features from the robotic arm control task based on historical interaction data to obtain task features;

[0015] The action and state determination module is used to determine the target robotic arm action based on the task characteristics through a preset strategy network;

[0016] The target action advantage calculation module is used to calculate the advantage value of the target robotic arm action through a preset advantage function to obtain the target action advantage;

[0017] The normalization module is used to normalize the target action advantage to obtain the target action weight;

[0018] The parameter update module is used to update the parameters of the preset policy network based on the target action weights to obtain the target policy network.

[0019] The robotic arm control module is used to control the robotic arm using a target policy network.

[0020] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0021] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0022] This application proposes a robotic arm control method, device, equipment, and medium based on meta-reinforcement learning. It extracts features from historical interaction data to generate task features representing task characteristics. These task features guide the policy network in selecting actions, improving the network's adaptability to different tasks. A pre-defined advantage function calculates the advantage of the target action, quantifying the quality of the target robotic arm action. The advantage of the target action is normalized and converted into weights. Based on these weights, the parameters of the pre-defined policy network are updated, ensuring that high-advantage actions receive high weights to dominate the training direction. Thus, the policy network can quickly adapt to unknown tasks with only a small number of samples, improving the generalization ability of the robotic arm control method. Attached Figure Description

[0023] Figure 1 This is a flowchart of the robotic arm control method based on meta-reinforcement learning provided in the embodiments of this application;

[0024] Figure 2 This is a flowchart illustrating the process of meta-reinforcement learning;

[0025] Figure 3 This is another flowchart of the robotic arm control method based on meta-reinforcement learning provided in the embodiments of this application;

[0026] Figure 4 This is a schematic diagram of the structure of the robotic arm control device based on meta-reinforcement learning provided in the embodiments of this application;

[0027] Figure 5 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0029] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0031] First, let's analyze some of the terms used in this application:

[0032] Meta-reinforcement learning (Meta-RL) is a machine learning method designed to train agents (such as AI agents) to quickly adapt to new tasks. By learning general learning policies across multiple tasks, it enables efficient decision-making in unseen environments. Meta-reinforcement learning is a subfield of artificial intelligence, an interdisciplinary field combining reinforcement learning and meta-learning, and is often referred to as meta-reinforcement learning or Meta-RL. Meta-reinforcement learning includes meta-policy optimization, model-independent meta-learning (MAML), and gradient-based adaptation algorithms. It is commonly used in fields such as robot control, game agent development, autonomous driving systems, resource management, and personalized recommendations.

[0033] Meta-training: Meta-training is the training phase in a meta-learning framework. It involves optimizing a meta-model (such as a neural network) across multiple related tasks to learn shared knowledge between tasks, enabling the model to generalize to new tasks. Meta-training is part of the machine learning process and is a core component of meta-learning algorithm design and implementation; it is often referred to as the meta-optimization phase. Meta-training includes steps such as task sampling, meta-loss function design, inner loop training, and outer loop updating.

[0034] Meta-testing is the evaluation phase in a meta-learning framework. It tests the generalization ability and adaptation efficiency of a meta-trained model on new tasks by simulating unknown environments to verify the model's performance. Meta-testing is part of the machine learning process and a crucial step in the validation and deployment of meta-learning; it is often referred to as the meta-evaluation phase. Meta-testing includes task adaptability testing, performance metrics (such as accuracy or reward scores), and robustness analysis.

[0035] In related technologies, see Figure 2 In offline meta-reinforcement learning, the tasks used in the meta-training and meta-testing phases differ. Meta-training uses task distributions built based on historical data or predefined task libraries, while the meta-testing phase handles completely new, unseen tasks. Consequently, tasks used in meta-testing do not appear in the meta-training tasks, leading to a distribution shift problem. This distribution shift problem results in insufficient generalization performance, preventing the agent from effectively fine-tuning with limited samples and resulting in poor generalization ability.

[0036] Based on this, embodiments of this application provide a robotic arm control method, apparatus, device, and medium based on meta-reinforcement learning, aiming to improve the generalization ability of the robotic arm control method.

[0037] The robotic arm control method, device, electronic device, and storage medium based on meta-reinforcement learning provided in this application are specifically described through the following embodiments. First, the robotic arm control method based on meta-reinforcement learning in this application embodiment is described.

[0038] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0039] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0040] The robotic arm control method based on meta-reinforcement learning provided in this application relates to the field of robotic arm control technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the meta-reinforcement learning-based robotic arm control method, but is not limited to the above forms.

[0041] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, multiprocessor systems, microprocessor-based systems, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0042] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0043] Figure 1 This is an optional flowchart of the robotic arm control method based on meta-reinforcement learning provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S107.

[0044] Step S101: Obtain historical interaction data of the robotic arm control task;

[0045] Step S102: Based on historical interaction data, feature extraction is performed on the robotic arm control task to obtain task features;

[0046] Step S103: Based on the task characteristics, determine the target robotic arm action through a preset strategy network;

[0047] Step S104: Calculate the advantage value of the target robotic arm's action using a preset advantage function to obtain the advantage of the target action;

[0048] Step S105: Normalize the target action advantage to obtain the target action weight;

[0049] Step S106: Based on the target action weights, update the parameters of the preset policy network to obtain the target policy network;

[0050] Step S107: Use the target policy network to control the robotic arm.

[0051] In a straightforward manner, historical interaction data includes trajectory data generated by the agent's interaction with the environment in multi-task scenarios, such as action-state-reward sequences. The target robotic arm action can include the action probability distribution output by the policy network (i.e., the probability value of selecting different actions), and the superiority or inferiority of each action is quantified through a preset advantage function. The advantage of the target action is normalized to obtain the target action weight. Based on the target action weight, the parameters of the preset policy network are updated to obtain the target policy network. In the meta-testing phase, the environmental state and the task characteristics of the new task are input into the target policy network, and the target policy network then guides the next robotic arm action to complete tasks such as grasping and stacking.

[0052] Steps S101 to S107 as illustrated in this embodiment of the application can generate task features that characterize the task by extracting features from historical interaction data; the task features guide the policy network to select actions, improving the policy network's adaptability to different tasks; the advantage of the target action is calculated by a preset advantage function, quantifying the quality of the target robotic arm action; the advantage of the target action is normalized and converted into weights, and the parameters of the preset policy network are updated based on the target action weights, ensuring that high-advantage actions receive high weights to dominate the training direction. Thus, the policy network can quickly adapt to unknown tasks with only a small number of samples, improving the generalization ability of the robotic arm control method.

[0053] The embodiments of this application can be applied to the motion control of various robotic arms and robots, such as industrial robotic arms deployed on assembly lines, robots that need to perform complex tasks, and robots deployed in space stations to complete related tasks. Taking robots performing complex tasks as an example, these robots need to perform a wide variety of tasks that are complex in program and highly automated, such as picking up certain items or unlocking doors. By applying the method described in the embodiments of this application, a certain amount of interaction data between the robot and the environment can be collected for training. Through the target policy network, the robot can complete similar tasks well, effectively making up for the shortcomings of traditional methods and reducing the cost of using robots.

[0054] In step S102 of some embodiments, the interactive data can be encoded by a preset inference network to make it conform to a Gaussian posterior distribution, and the posterior distribution parameters z (such as mean and variance) of the robotic arm control task can be obtained as task features.

[0055] In some embodiments, the meta-reinforcement learning-based robotic arm control method further includes: setting the loss function of the inference network as... Where β represents the regularization coefficient, Let p(z) represent the inference network, which outputs the posterior distribution parameter z based on the input data (i.e., the interaction data c), where p(z) = N(0, I). In this embodiment, the information content of the latent posterior distribution parameter z is limited by KL divergence, essentially constraining the inference network. The difference between the output data and the prior distribution p(z) (usually assumed to be a normal distribution N(0,I)) if If the output data carries too much information that is irrelevant to the task, the KL value will increase. By minimizing the KL divergence, the distribution of z is forced to be close to a simple prior distribution, thereby limiting its information content and preventing overfitting of the extracted latent posterior distribution parameter z due to insufficient data.

[0056] In some embodiments, step S103 may include, but is not limited to, steps S301 to S302:

[0057] Step S301: Obtain the initial environment state;

[0058] Step S302: Based on the task characteristics and initial environmental state, select actions through a preset strategy network to obtain the target robotic arm action.

[0059] In a straightforward manner, the initial environmental state can include the robotic arm's posture, position, and object position. The target robotic arm action can be the motion parameters of each joint of the robotic arm. After executing the target robotic arm action, the environmental state can be updated based on environmental feedback to obtain the target environmental state.

[0060] In some embodiments, step S104 may include, but is not limited to, steps S401 to S403:

[0061] Step S401: Calculate the value of the target robotic arm's motion using a preset motion value network to obtain the target motion value;

[0062] Step S402: Calculate the value of the initial environmental state through a preset state value network to obtain the initial state value;

[0063] Step S403: Based on the target action value and the initial state value, calculate the advantage value of the target robotic arm action using a preset advantage function to obtain the target action advantage.

[0064] In step S403 of some embodiments, the difference between the target action value and the initial state value is taken as the target action advantage. Specifically, the advantage function is set as A(s,a) = Q(s,a) - V(s), where Q(s,a) is the target action value output by the action value network (i.e., the Q-value network), and V(s) is the initial state value output by the state value network (i.e., the V-value network).

[0065] In some embodiments, the robotic arm control method based on meta-reinforcement learning further includes steps S501 to S506:

[0066] Step S501: Update the environmental state based on the target robotic arm's movements and the initial environmental state to obtain the target environmental state;

[0067] Step S502: Calculate the average value of the task features to obtain the mean value of the task features;

[0068] Step S503: Based on the mean of the task features, calculate the value of the target environment state to obtain the target state value;

[0069] Step S504: Obtain the reward for the target robotic arm's action;

[0070] Step S505: Calculate the action value loss based on the target action value, reward, and target state value;

[0071] Step S506: Update the parameters of the action value network based on the action value loss.

[0072] In some embodiments, the robotic arm control method based on meta-reinforcement learning further includes steps S601 to S602:

[0073] Step S601: Calculate the state value loss based on the target action value and the initial state value;

[0074] Step S602: Update the parameters of the state value network based on the state value loss.

[0075] Specifically, refer to Figure 3 The system collects trajectory data generated by the interaction between the agent and the environment when performing different tasks, such as Markov decision tuples (action-state-reward). Then, it uses probabilistic modeling to model the Markov decision process, obtaining the task distribution p(T), T = {S, A, T, R}, where S is the state space, A is the action space, T is the state transition probability, and R is the reward function. The system then randomly samples the trajectory data T of the i-th task from the task distribution p(T). i Then store it in the experience replay buffer. Historical interaction data is used as the basis for feature extraction. This historical interaction data is processed using a Transformer model to generate task features z. Based on the task features z and the current policy, the policy network (Actor) outputs the action probability distribution. The action value network (Critic) contains two Q-value networks (Q1 and Q2), which output the action value (Q-value) based on the task features z and the action probability distribution. The state value network (Value Network) outputs the state value (V-value) based on the state.

[0076] Set the V-value loss function to calculate the state value loss. In the formula, V(s,z) represents the initial state value; Q θ (s,a,z) represents the target action value; E (s,a,r,s′) This represents the expectation of the joint distribution of the initial state, action, reward, and target state.

[0077] Set the Q-value loss function to calculate the action value loss. in, Let Q represent the expectation of the joint distribution of the initial state, action, reward, and target state sampled from the experience replay buffer; θ(s,a,z) represents the value of the target action; r represents the reward. For the target state value, represents the average feature value of multiple sampled tasks; s and s′ represent the current state (i.e., the initial environment state) and the next state (i.e., the target environment state), respectively; a is the current action (i.e., the target robotic arm action). During training, the goal is to minimize the action value loss, making Q... θ The optimal value function is gradually approximated.

[0078] In this embodiment, the V-value network parameters are updated by using the environmental state s and the latent posterior distribution parameter z, and the Q-value network parameters are updated by using the output of the V-value network. This ensures the conservatism of the Q-value network estimation and avoids performance degradation and errors caused by overestimating the value of actions.

[0079] In step S105 of some embodiments, the target action advantage can be normalized by softmax to obtain the target action weight ω(s,a).

[0080] In step S106 of some embodiments, the target action weight can be multiplied by the action probability distribution output by the policy network to obtain a weighted action probability distribution; based on the weighted action probability distribution, the policy loss weight is calculated, and the policy loss weight is minimized by gradient descent to obtain the target policy network.

[0081] In some embodiments, step S106 may include, but is not limited to, steps S701 to S705:

[0082] Step S701: Calculate the weighted value of the probability distribution of the target robotic arm's actions based on the target action weights to obtain the first distribution;

[0083] Step S702: Calculate the motion value of the target robotic arm's motion to obtain the target motion value;

[0084] Step S703: Calculate the probability distribution of the target action value to obtain the second distribution;

[0085] Step S704: Calculate the difference between the first distribution and the second distribution to obtain the policy loss;

[0086] Step S705: Based on the policy loss, update the parameters of the policy network to obtain the target policy network.

[0087] In simple terms, the first distribution represents the probability distribution of different actions taken by the policy network under the current policy parameters for a specific state and task characteristics. The second distribution represents the probability distribution of different action values ​​under a given state and task characteristics, characterizing the optimal distribution of action values. The policy loss measures the performance of the policy network by calculating the difference between the first and second distributions, and updates the parameters of the policy network by minimizing the policy loss. This allows the policy network to quickly adapt and fine-tune when facing new tasks, thereby improving the operational performance and efficiency of the robotic arm in complex environments.

[0088] Steps S701 to S705 as illustrated in this embodiment involve weighting the probability distribution of the target robotic arm's actions based on the target action weights to generate a first distribution, assigning higher weights to high-advantage actions to highlight their contribution; calculating the action value of the target robotic arm's actions to obtain the target action value, quantifying the expected cumulative return of the action; using the probability distribution of the target action value as a second distribution; and generating a strategy loss by calculating the difference between the first and second distributions. The strategy loss reflects the deviation between the current strategy network output and the optimal value distribution, which can transform action advantage information into a directional optimization signal for the strategy network, forcing the strategy network parameters to be updated in the direction of increasing probability of high-value actions. This enables a more accurate action selection strategy, accelerates strategy network convergence, and improves sample utilization efficiency in robotic arm control tasks.

[0089] In step S704 of some embodiments, the policy loss is set:

[0090]

[0091] in, For strategic losses; This is used to ensure that the product of the target action weight ω(s,a) and the policy network π(a|s) forms an effective probability distribution in the action space.

[0092] In step S705 of some embodiments, based on policy loss The parameters of the policy network are updated using gradient descent. After the meta-training phase, the trained policy network can be used for fine-tuning on new, unseen tasks during the meta-testing phase.

[0093] Please see Figure 4 This application also provides a robotic arm control device based on meta-reinforcement learning, which can implement the above-mentioned robotic arm control method based on meta-reinforcement learning. The device includes:

[0094] The interactive data acquisition module is used to acquire historical interactive data of the robotic arm control tasks;

[0095] The task feature extraction module is used to extract features from the robotic arm control task based on historical interaction data to obtain task features;

[0096] The action and state determination module is used to determine the target robotic arm action based on the task characteristics through a preset strategy network;

[0097] The target action advantage calculation module is used to calculate the advantage value of the target robotic arm action through a preset advantage function to obtain the target action advantage;

[0098] The normalization module is used to normalize the target action advantage to obtain the target action weight;

[0099] The parameter update module is used to update the parameters of the preset policy network based on the target action weights to obtain the target policy network.

[0100] The robotic arm control module is used to control the robotic arm using a target policy network.

[0101] The specific implementation of the meta-reinforcement learning-based robotic arm control device is basically the same as the specific implementation of the meta-reinforcement learning-based robotic arm control method described above, and will not be repeated here.

[0102] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described robotic arm control method based on meta-reinforcement learning. This electronic device can be any intelligent terminal such as an in-vehicle computer.

[0103] Please see Figure 5 , Figure 5 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0104] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), GPU (Graphics Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0105] The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 to implement the meta-reinforcement learning-based robotic arm control method of the embodiments of this application.

[0106] The input / output interface 903 is used to implement information input and output;

[0107] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0108] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0109] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0110] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described robotic arm control method based on meta-reinforcement learning.

[0111] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0112] The robotic arm control method, device, electronic device, and storage medium based on meta-reinforcement learning provided in this application can generate task features that characterize task properties by extracting features from historical interaction data. These task features guide the policy network in selecting actions, improving the network's adaptability to different tasks. A preset advantage function is used to calculate the advantage of the target action, quantifying the quality of the target robotic arm action. The advantage of the target action is normalized and converted into weights. Based on these weights, the parameters of the preset policy network are updated, ensuring that high-advantage actions receive high weights to dominate the training direction. Thus, the policy network can quickly adapt to unknown tasks with only a small number of samples, improving the generalization ability of the robotic arm control method.

[0113] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0114] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0115] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0116] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0117] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0118] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0119] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0120] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0121] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0122] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0123] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A robotic arm control method based on meta-reinforcement learning, characterized in that, The method includes: Acquire historical interaction data of robotic arm control tasks; Based on the historical interaction data, feature extraction is performed on the robotic arm control task to obtain task features; Based on the task characteristics, the target robotic arm action is determined through a preset strategy network; The advantage of the target action is obtained by calculating the advantage value of the target robotic arm action through a preset advantage function; The target action advantage is normalized to obtain the target action weight; Based on the target action weights, the parameters of the preset policy network are updated to obtain the target policy network; The robotic arm is controlled using the target policy network.

2. The method according to claim 1, characterized in that, The step of updating the parameters of the preset policy network based on the target action weights to obtain the target policy network includes: Based on the target action weight, the weighted value of the probability distribution of the target robotic arm action is calculated to obtain the first distribution; Calculate the motion value of the target robotic arm's action to obtain the target motion value; Calculate the probability distribution of the value of the target action to obtain the second distribution; Calculate the difference between the first distribution and the second distribution to obtain the policy loss; Based on the policy loss, the parameters of the policy network are updated to obtain the target policy network.

3. The method according to claim 1, characterized in that, The step of determining the target robotic arm action based on the task characteristics through a preset policy network includes: Obtain the initial environment state; Based on the task characteristics and the initial environmental state, the target robotic arm action is obtained by selecting an action through a preset strategy network.

4. The method according to claim 3, characterized in that, The step of calculating the advantage value of the target robotic arm's movement through a preset advantage function to obtain the target movement advantage includes: The target action value is obtained by calculating the value of the target robotic arm action through a preset action value network; The value of the initial environment state is calculated by using a preset state value network to obtain the initial state value. Based on the target action value and the initial state value, the advantage value of the target robotic arm action is calculated using a preset advantage function to obtain the target action advantage.

5. The method according to claim 4, characterized in that, The method further includes: Based on the target robotic arm's movements and the initial environmental state, the environmental state is updated to obtain the target environmental state; Calculate the average value of the task features to obtain the mean value of the task features; Based on the average of the task characteristics, the value of the target environment state is calculated to obtain the target state value; Obtain a reward for the target robotic arm's action; Based on the target action value, the reward, and the target state value, the action value loss is calculated. Based on the loss of action value, the parameters of the action value network are updated.

6. The method according to claim 4, characterized in that, The method further includes: Based on the target action value and the initial state value, the state value loss is calculated; Based on the state value loss, the parameters of the state value network are updated.

7. The method according to claim 1, characterized in that, Based on the historical interaction data, feature extraction is performed on the robotic arm control task to obtain task features, including: The historical interaction data is encoded using a pre-defined inference network to obtain the posterior distribution parameters of the robotic arm control task as the task features.

8. A robotic arm control device based on meta-reinforcement learning, characterized in that, The device includes: The interactive data acquisition module is used to acquire historical interactive data of the robotic arm control tasks; The task feature extraction module is used to extract features from the robotic arm control task based on the historical interaction data to obtain task features; The action and state determination module is used to determine the target robotic arm action based on the task characteristics through a preset strategy network; The target action advantage calculation module is used to calculate the advantage value of the target robotic arm action through a preset advantage function to obtain the target action advantage; The normalization processing module is used to normalize the target action advantage to obtain the target action weight; The parameter update module is used to update the parameters of the preset policy network based on the target action weights to obtain the target policy network. A robotic arm control module is used to control the robotic arm using the target policy network.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the robotic arm control method based on meta-reinforcement learning as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the robotic arm control method based on meta-reinforcement learning as described in any one of claims 1 to 7.

Images